Hello all,
I apologize for joining this conversation so late; I am a longtime, though recently quiet, member of the community.
I understand that the ranking system is an attempt to enable people to judge the quality of a contribution (or contributor) based on some directly observable measure of reputation. On first sight, one could argue that such a system overcomes the problem that in the face-to-face world reputation is not directly observable. In order to get an honest assessment of a person's reputation, one has to invest a lot of time building trust with the people familiar with that person. Even then, a trustworthy assessment requires direct observation of the person's actions.
A ranking system would appear to reduce that effort by half; knowing the person's reputation among his/her peers, one only needs to assess the work.
The problem with that reasoning is that electronic ranking systems are highly susceptible to manipulation. Building reputation becomes the goal of the activity for many people and they use all sorts of seemingly harmless social and technological means to inflate their numbers. Our lab has been studying this phenomenon through both observational studies of online communities and laboratory experiments. The two papers below report on the phenomenon as it presents itself in Digg, the news aggregation site.
The first paper demonstrates that a tit-for-tat game of reciprocity inflates the reputation of contributors and their contributions without reflecting anything substantive about their contributions. The second paper really brings out the negative consequences of this phenomenon. The paper reports on an experiment where people judged how interesting they found the contribution. The ranking values of the articles were set by the investigator; sometime the rank of the article was set high, at other times low. Experimental subjects rated higher-ranked contributions as more interesting than lower ranked contributions. The same article was rated as highly interesting when its rank was set high and uninteresting when ranked low. Duncan Watts (of small-world networks fame) observed the same phenomenon with music rating.
What this means in the present discussion is that people will likely ignore low ranking contributions. Worse still, when they do actually look at those contributions they are likely to see what the ranking value led them to expect rather than the qualities of the contribution itself.
Unless we can find scientific research that demonstrates any benefits to ranking, I think we should be wary of using such systems.
All best,
John
Sadlon, E., Sakamoto, Y., Dever, H. J., & Nickerson, J. V. (2008). The karma of Digg: Reciprocity in online social networks. In R. Gopal and R. Ramesh (Eds.), Proceedings of the 18th Annual Workshop on Information Technologies and Systems. http://cog.mgnt.stevens-tech.edu/~yasu/papers/reciprocity.pdf
Sakamoto, Y., Ma, J., & Nickerson, J. V. (2009). 2377 people like this article: The influence of others' decisions on yours. In N. Taatgen, H. van Rijn, L. Schomaker, and J. Nerbonne (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society. http://cog.mgnt.stevens-tech.edu/~yasu/papers/cogscidigg1.pdf
Salganik, M. J., Dodds, P. S., and Watts, D. J. (2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311(5762):854-856. http://dx.doi.org/10.1126/science.1121066%5B/url]
-------------------- m2f --------------------
(from forum) http://squeakland.org/forums/viewtopic.php?p=11554#11554
-------------------- m2f --------------------
On Sep 30, 2009, at 10:58 AM, voiklis wrote:
Hello all,
I apologize for joining this conversation so late; I am a longtime, though recently quiet, member of the community.
I understand that the ranking system is an attempt to enable people to judge the quality of a contribution (or contributor) based on some directly observable measure of reputation. On first sight, one could argue that such a system overcomes the problem that in the face-to-face world reputation is not directly observable. In order to get an honest assessment of a person's reputation, one has to invest a lot of time building trust with the people familiar with that person. Even then, a trustworthy assessment requires direct observation of the person's actions.
A ranking system would appear to reduce that effort by half; knowing the person's reputation among his/her peers, one only needs to assess the work.
The problem with that reasoning is that electronic ranking systems are highly susceptible to manipulation. Building reputation becomes the goal of the activity for many people and they use all sorts of seemingly harmless social and technological means to inflate their numbers. Our lab has been studying this phenomenon through both observational studies of online communities and laboratory experiments. The two papers below report on the phenomenon as it presents itself in Digg, the news aggregation site.
The first paper demonstrates that a tit-for-tat game of reciprocity inflates the reputation of contributors and their contributions without reflecting anything substantive about their contributions. The second paper really brings out the negative consequences of this phenomenon. The paper reports on an experiment where people judged how interesting they found the contribution. The ranking values of the articles were set by the investigator; sometime the rank of the article was set high, at other times low. Experimental subjects rated higher-ranked contributions as more interesting than lower ranked contributions. The same article was rated as highly interesting when its rank was set high and uninteresting when ranked low. Duncan Watts (of small-world networks fame) observed the same phenomenon with music rating.
What this means in the present discussion is that people will likely ignore low ranking contributions. Worse still, when they do actually look at those contributions they are likely to see what the ranking value led them to expect rather than the qualities of the contribution itself.
Unless we can find scientific research that demonstrates any benefits to ranking, I think we should be wary of using such systems.
Thank you for your insightful response.
A few things that may make the Squeakland ranking system different from those you cite . .
1) viewers don't rank, so it's not a popularity contest
2) only people with established credibility rank projects ... you've got to earn your say
3) ranking is limited to ten people per project
4) rankers are chosen at random, with bias to higher credibility ... so friends are less likely to rank each other
5) number or value of votes isn't shown anywhere, it's merely the position in the list
6) there are many ways to view projects (by group, by tag, by subject, by region) so you're much more likely to see lower ranked projects than in a straight sort
Anyway, these are just off the top of my head. The benefits to the ranking system lie chiefly in:
1) letting people see which accounts have long-term credibility, which I've found useful on other sites
2) distributing the ranking effort while maintaining quality by favoring those with credibility
3) showing the best projects first, in a uniform & fair way, which is what it's all about
It's really more of a trust system than a ranking system. It's just that ranking is the chief way to get trust.
Tim
squeakland@lists.squeakfoundation.org