Thumbs-Up: a game for playing to rank search results

ABSTRACT

Human computation is an effective way to channel human effort spent playing games to solving computational problems that are easy for humans but difficult for computers to automate. We propose Thumbs-Up, a new game for human computation with the purpose of playing to rank search result. Our experience from users shows that Thumbs-Up is not only fun to play, but produces more relevant rankings than both a major search engine and optimal rank aggregation using the well-known Kemeny rule.

REFERENCES

• 1: P. N. Bennett, D. M. Chickering, and A. Mityagin. Learning consensus opinion: Mining data from a labeling game. In Proc. Int. Conf. on WWW, pp. 121--30, ACM, 2009.
• 2: B. Carterette, P. Bennett, D. M. Chichering, and S. Dumais. Here or there: Preference judgments for relevance. In Proc. Int. Euro. Conf. on IR, pp. 16--27. ACM, 2008.
• 3: A. Dasdan, C. Drome, and S. Kolay. Thumbs-Up: A Game for Playing to Rank Search Results In Proc. Int. Conf. on WWW, pp. 1071--2, ACM, 2009.
• 4: C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proc. Int. Conf. on WWW, pp. 613--22. ACM, 2001.
• 5: The Entertainment Software Association: Industry facts. http://www.theesa.com/facts/, Dec 2008.
• 6: S.Hacker and L. von Ahn. Matchin: Eliciting user preferences with an online game. In Proc. Int. Conf. on CHI, pp. 1207--16. ACM, 2009.
• 7: K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422--46, 2002.
• 8: L. von Ahn and L. Dabbish. Designing games with a purpose. Commun. ACM, 51(8):58--67, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
games with a purpose, human computation, online games, rank aggregation, relevance, search engine