Input-agreement: a new mechanism for collecting data using human computation games

ABSTRACT

Since its introduction at CHI 2004, the ESP Game has inspired many similar games that share the goal of gathering data from players. This paper introduces a new mechanism for collecting labeled data using "games with a purpose." In this mechanism, players are provided with either the same or a different object, and asked to describe that object to each other. Based on each other's descriptions, players must decide whether they have the same object or not. We explain why this new mechanism is superior for input data with certain characteristics, introduce an enjoyable new game called "TagATune" that collects tags for music clips via this mechanism, and present findings on the data that is collected by this game.

REFERENCES

• 1: Severin Hacker , Luis von Ahn, Matchin: eliciting user preferences with an online game, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA [doi>10.1145/1518701.1518882]
• 2: Healey, P., Swoboda, N., Umata, I., and King, J. Graphical language games: interactional constraints on representational form. Cognitive Science, 31:285--309, 2007.
• 3: Kim, Y.E., Schmidt, E., and Emelle, L. MoodSwings: A collaborative game for music mood label collection. In Proc. 9th Intl. Conf. on Music Information Retrieval (Philadelphia, September 14-18). 231--236, 2008.
• 4: Lamere, P. Social tagging and music information retrieval. Journal of New Music Research. 37(2):101--104, 2008.
• 5: Law, E., von Ahn L., Dannenberg, R., and Crawford M. TagATune: a game for music and sound annotation. In Proc. 8th Intl. Conf. on Music Information Retrieval (Vienna, September 23-27). 361--364, 2007.
• 6: Law, E., Mityagin, A., and Chickering, M. Intentions: A game for classifying search query intent. In submission.
• 7: Lee, B. and von Ahn, L. Squigl: A Web game to generate datasets for object detection algorithms. In submission.
• 8: Ma, H., Gupta, A., and Chandrasekar, R. Page hunt, page race and page match: using competitive and collaborative games to improve Web search and understand user behavior. In submission.
• 9: Mandel, M. and Ellis, D. A Web-based game for collecting music metadata. Journal of New Music Research. 37(2):151--165, 2009.
• 10: Mityagin, A. and Chickering, M. PictureThis. http://club.live.com/Pages/Games/GameList.aspx?game=Picture_This
• 11: Katharina Siorpaes , Martin Hepp, Games with a Purpose for the Semantic Web, IEEE Intelligent Systems, v.23 n.3, p.50-60, May 2008 [doi>10.1109/MIS.2008.45]
• 12: Turnbull, D., Liu, R., Barrington, L., and Lanckriet, G. A game-based approach for collecting semantic annotations of music. In Proc. 8th Intl. Conf. on Music Information Retrieval (Vienna, September 23-27). 535--538, 2007.
• 13: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]
• 14: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 15: Luis von Ahn , Laura Dabbish, Designing games with a purpose, Communications of the ACM, v.51 n.8, August 2008 [doi>10.1145/1378704.1378719]
• 16: Weinberger, D. How tagging changes peoples relationship to information and each other. Pew Internet&American Life Project. http://www.pewinternet.org/ pdfs/PIP_Tagging.pdf

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

General Terms:
Design

Keywords:
games with a purpose, human computation, tagging