Matchin: eliciting user preferences with an online game

ABSTRACT

Eliciting user preferences for large datasets and creating rankings based on these preferences has many practical applications in community-based sites. This paper gives a new method to elicit user preferences that does not ask users to tell what they prefer, but rather what a random person would prefer, and rewards them if their prediction is correct. We provide an implementation of our method as a two-player game in which each player is shown two images and asked to click on the image their partner would prefer. The game has proven to be enjoyable, has attracted tens of thousands of people and has already collected millions of judgments. We compare several algorithms for combining these relative judgments between pairs of images into a total ordering of all images and present a new algorithm to perform collaborative filtering on pair-wise relative judgments. In addition, we show how merely observing user preferences on a specially chosen set of images can predict a user's gender with high probability.

REFERENCES

• 1: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 2: Edith Law , Luis von Ahn, Input-agreement: a new mechanism for collecting data using human computation games, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA [doi>10.1145/1518701.1518881]
• 3: Lee, B. and von Ahn, L. Squigl: A Web game to generate datasets for object detection algorithms. In submission.
• 4: Luis von Ahn , Ruoran Liu , Manuel Blum, Peekaboom: a game for locating objects in images, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124782]
• 5: Luis von Ahn , Mihir Kedia , Manuel Blum, Verbosity: a game for collecting common-sense facts, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124784]
• 6: Flickr, a Web site for photo-sharing. http://www.flickr.com
• 7: Butterfield; Daniel S.; et al. Interestingness ranking of media objects. U.S. Patent application 2006/0242139 A1, filed February 8, 2006.
• 8: Digg, a Web site for social bookmarking. http://www.digg.com
• 9: Youtube, a Web site for sharing videos. http://www.youtube.com
• 10: Hot or Not, a Web site for rating pictures of people. http://www.hotornot.com
• 11: Krzysztof Gajos , Daniel S. Weld, Preference elicitation for interface optimization, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095063]
• 12: GWAP, a Web site for Games with a Purpose. http://www.gwap.com
• 13: Elo, Arpad. The Rating of Chessplayers, Past and Present. Arco Publications, New York, 1978.
• 14: Herbrich, R., Minka, T., and Graepel, T. TrueSkillTM: A Bayesian Skill Rating System. In Proc. Advances in Neural Information Processing Systems 19, MIT Press, Cambridge, MA, 2007, 569--576.
• 15: G. Takács, I. Pilászy, B. Németh, and D. Tikk. On the Gravity Recommendation System. In Proc. 13th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (San Jose, August 12-15). ACM, New York, 2007, 22--30.
• 16: Amazon Mechanical Turk. http://www.mturk.com

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

General Terms:
Experimentation, Human Factors

Keywords:
human computation, preference elicitation, web games