Thumbs-up: a game for playing to rank search results

ABSTRACT

Human computation is an effective way to channel human effort spent playing games to solving computational problems that are easy for humans but difficult for computers to automate. We propose Thumbs-Up, a new game for human computation with the purpose of playing to rank search result. Our experience from users shows that Thumbs-Up is not only fun to play, but produces more relevant rankings than both a major search engine and optimal rank aggregation using the Kemeny rule.

REFERENCES

• 1: B. Carterette, P. Bennett, D. M. Chichering, and S. Dumais. Here or there: Preference judgments for relevance. In Proc. Int. Euro. Conf. on IR (ECIR), pp. 16--27. ACM, 2008.
• 2: Cynthia Dwork , Ravi Kumar , Moni Naor , D. Sivakumar, Rank aggregation methods for the Web, Proceedings of the 10th international conference on World Wide Web, p.613-622, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372165]
• 3: The Entertainment Software Association: Industry facts. http://www.theesa.com/facts/, Dec 2008.
• 4: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 5: Luis von Ahn , Laura Dabbish, Designing games with a purpose, Communications of the ACM, v.51 n.8, August 2008 [doi>10.1145/1378704.1378719]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Knowledge acquisition

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
games with a purpose, human computation, online games, rank aggregation, relevance, search engine