Labeling images with a computer game

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Labeling images with a computer game

Full text

Pdf (494 KB)

Source

Conference on Human Factors in Computing Systems archive
Proceedings of the SIGCHI conference on Human factors in computing systems table of contents

Vienna, Austria

Pages: 319 - 326

Year of Publication: 2004

ISBN:1-58113-702-8

Authors

Luis von Ahn	Carnegie Mellon University, Pittsburgh, PA
Laura Dabbish	Carnegie Mellon University, Pittsburgh, PA

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction
SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web
ACM: Association for Computing Machinery
SIGCAPH: ACM SIGCAPH Computers and the Physically Handicapped
SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
SIGGROUP: ACM Special Interest Group on Supporting Group Work
SIGDOC : ACM Special Interest Group on Systems Documentation

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 41, Downloads (12 Months): 287, Citation Count: 102

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/985692.985733 What is a DOI?

ABSTRACT

We introduce a new interactive system: a game that is fun and can be used to create valuable output. When people play the game they help determine the contents of images by providing meaningful labels for them. If the game is played as much as popular online games, we estimate that most images on the Web can be labeled in a few months. Having proper labels associated with each image on the Web would allow for more accurate image search, improve the accessibility of sites (by providing descriptions of images to visually impaired individuals), and help users block inappropriate images. Our system makes a significant contribution because of its valuable output and because of the way it addresses the image-labeling problem. Rather than using computer vision techniques, which don't work well enough, we encourage people to do the work by taking advantage of their desire to be entertained.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	AltaVista Company. Altavista Search Home. http://www.altavista.com/
	2	Barnard, K., Duygulu, P., and Forsyth, D. A. Clustering Art. IEEE conference on Computer Vision and Pattern Recognition, 2001, pages 434--441.
	3	Barnard, K., and Forsyth, D. A. Learning the Semantics of Words and Pictures. International Conference of Computer Vision, 2001, pages 408--415.
	4	Carson, C., and Ogle, V. E. Storage and Retrieval of Feature Data for a Very Large Online Image Collection <http://elib.cs.berkeley.edu/papers/dataeng.html>. IEEE Computer Society Bulletin of the Technical Committee on Data Engineering, 1996, Vol. 19 No. 4.
	5	Columbia University Center for Telecommunications Research. webSEEK. www.ctr.columbia.edu/webseek/
	6	Corbis. Stock Photography and Pictures. http://www.corbis.com/
	7	P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
	8	Electronic Arts. Pogo.com. http://www.pogo.com
	9	Margaret M. Fleck , David A. Forsyth , Chris Bregler, Finding Naked People, Proceedings of the 4th European Conference on Computer Vision-Volume II, p.593-602, April 15-18, 1996
	10	Google Inc. Google Web Search. http://www.google.com
	11	Ronny Lempel , Aya Soffer, PicASHOW: pictorial authority search by hyperlinks on the Web, Proceedings of the 10th international conference on World Wide Web, p.438-448, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372098]
	12	Microsoft Corporation. MSN Gaming Zone. http://zone.msn.com
	13	Milliman, R. E. Website Accessibility And The Private Sector. www.rit.edu/~easi/itd/itdv08n2/milliman.html
	14	N2H2, Inc. N2H2 Filter. http://www.n2h2.com
	15	O'Connor, B. and O'Connor, M. Categories, Photographs & Predicaments: Exploratory Research on Representing Pictures for Access. Bulletin of the American Society for Information Science 25.6, 1999, page 17--20.
	16	Random Bounce Me Website. http://random.bounceme.net/
	17	Henry Schneiderman , Takeo Kanade, Object Detection Using the Statistics of Parts, International Journal of Computer Vision, v.56 n.3, p.151-177, February-March 2004 [doi>10.1023/B:VISI.0000011202.85607.00]
	18	Stork, D. G. and Lam C. P. Open Mind Animals: Ensuring the quality of data openly contributed over the World Wide Web. AAAI Workshop on Learning with Imbalanced Data Sets, 2000, pages 4--9.
	19	Stork, D. G. The Open Mind Initiative. IEEE Intelligent Systems & Their Applications, 14-3, 1999, pages 19--20.
	20	Trask, R. L. Language Change. Routledge, 1994
	21	Yahoo!, Inc. Yahoo! Games. http://games.yahoo.com

CITED BY 102

	Luis von Ahn , Shiry Ginosar , Mihir Kedia , Ruoran Liu , Manuel Blum, Improving accessibility of the web with a computer game, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Matthew Chalmers, A Historical View of Context, Computer Supported Cooperative Work, v.13 n.3-4, p.223-247, August 2004
	Rebecca E. Grinter, Words about Images: Coordinating Community in Amateur Photography, Computer Supported Cooperative Work, v.14 n.2, p.161-188, April 2005
	Daniel H. Wilson , Anna C. Long , Chris Atkeson, A context-aware recognition survey for data collection using ubiquitous sensors in the home, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA
	Craig Gentry , Zulfikar Ramzan , Stuart Stubblebine, Secure distributed human computation, Proceedings of the 6th ACM conference on Electronic commerce, p.155-164, June 05-08, 2005, Vancouver, BC, Canada
	Jack Kustanowitz , Ben Shneiderman, Meaningful presentations of photo libraries: rationale and applications of bi-level radial quantum layouts, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA
	Luis von Ahn , Mihir Kedia , Manuel Blum, Verbosity: a game for collecting common-sense facts, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Timothy Chklovski, Collecting paraphrase corpora from volunteer contributors, Proceedings of the 3rd international conference on Knowledge capture, October 02-05, 2005, Banff, Alberta, Canada
	Ritendra Datta , Jia Li , James Z. Wang, Content-based image retrieval: approaches and trends of the new age, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore
	Luis von Ahn , Ruoran Liu , Manuel Blum, Peekaboom: a game for locating objects in images, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Ben Shneiderman , Benjamin B. Bederson , Steven M. Drucker, Find that photo!: interface strategies to annotate, browse, and share, Communications of the ACM, v.49 n.4, April 2006
	Timo Volkmer , John R. Smith , Apostol (Paul) Natsev, A web-based system for collaborative annotation of large image and video collections: an evaluation and user study, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore
	Dan Cosley , Dan Frankowski , Loren Terveen , John Riedl, Using intelligent task routing and contribution review to help communities build artifacts of lasting value, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	George W. Furnas , Caterina Fake , Luis von Ahn , Joshua Schachter , Scott Golder , Kevin Fox , Marc Davis , Cameron Marlow , Mor Naaman, Why do tagging systems work?, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada
	Alon Halevy , Michael Franklin , David Maier, Principles of dataspace systems, Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, p.1-9, June 26-28, 2006, Chicago, IL, USA
	Bryan C. Russell , Antonio Torralba , Kevin P. Murphy , William T. Freeman, LabelMe: A Database and Web-Based Tool for Image Annotation, International Journal of Computer Vision, v.77 n.1-3, p.157-173, May 2008
	Rong Yan , Apostol Natsev , Murray Campbell, An efficient manual image annotation approach based on tagging and browsing, Workshop on multimedia information retrieval on The many faces of multimedia semantics, September 28-28, 2007, Augsburg, Bavaria, Germany
	Cheng-Lun Li , Ayse G. Buyuktur , David K. Hutchful , Natasha B. Sant , Satyendra K. Nainwal, Portalis: using competitive online interactions to support aid initiatives for the homeless, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Ryan Shaw , Patrick Schmitz, Community annotation and remix: a research platform and pilot deployment, Proceedings of the 1st ACM international workshop on Human-centered multimedia, October 27-27, 2006, Santa Barbara, California, USA
	Satanjeev Banerjee , Alexander I. Rudnicky, Segmenting meetings into agenda items by extracting implicit supervision from human note-taking, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA
	Jeffrey P. Bigham , Ryan S. Kaminsky , Richard E. Ladner , Oscar M. Danielsson , Gordon L. Hempton, WebInSight:: making web images accessible, Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, October 23-25, 2006, Portland, Oregon, USA
	David A. Shamma , Bryan Pardo, Karaoke callout: using social and collaborative cell phone networking for new entertainment modalities and data collection, Proceedings of the 1st ACM workshop on Audio and music computing multimedia, October 27-27, 2006, Santa Barbara, California, USA
	Alon Halevy , Anand Rajaraman , Joann Ordille, Data integration: the teenage years, Proceedings of the 32nd international conference on Very large data bases, September 12-15, 2006, Seoul, Korea
	Cameron Marlow , Mor Naaman , Danah Boyd , Marc Davis, HT06, tagging paper, taxonomy, Flickr, academic article, to read, Proceedings of the seventeenth conference on Hypertext and hypermedia, August 22-25, 2006, Odense, Denmark
	Prabhakar Raghavan, The changing face of web search: algorithms, auctions and advertising, Proceedings of the thirty-eighth annual ACM symposium on Theory of computing, May 21-23, 2006, Seattle, WA, USA
	Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
	Shilad Sen , Shyong K. Lam , Al Mamunur Rashid , Dan Cosley , Dan Frankowski , Jeremy Osterhouse , F. Maxwell Harper , John Riedl, tagging, communities, vocabulary, evolution, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada
	Cosmin Munteanu , Gerald Penn , Ron Baecker , Yuecheng Zhang, Automatic speech recognition for webcasts: how good is good enough and what to do when it isn't, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada
	Jeffrey P. Bigham, Increasing web accessibility by automatically judging alternative text quality, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA
	Nilesh Dalvi , Dan Suciu, Management of probabilistic data: foundations and challenges, Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, p.1-12, June 11-13, 2007, Beijing, China
	Jingyu Cui , Fang Wen , Rong Xiao , Yuandong Tian , Xiaoou Tang, EasyAlbum: an interactive photo annotation system based on face clustering and re-ranking, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA
	Cosmin Munteanu , Ron Baecker , Gerald Penn, Collaborative editing for improved usefulness and usability of transcript-enhanced webcasts, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Börkur Sigurbjörnsson , Roelof van Zwol, Flickr tag recommendation based on collective knowledge, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China
	Rong Yan , Apostol Natsev , Murray Campbell, Multi-query interactive image and video retrieval -: theory and practice, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada
	Kenneth Treharne , Darius Pfitzner , Richard Leibbrandt , David Powers, A lean online approach to human factors research, Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments, July 16-18, 2008, Athens, Greece
	Shravan Gaonkar , Jack Li , Romit Roy Choudhury , Landon Cox , Al Schmidt, Micro-Blog: sharing and querying content through mobile phones and social participation, Proceeding of the 6th international conference on Mobile systems, applications, and services, June 17-20, 2008, Breckenridge, CO, USA
	Ed H. Chi , Todd Mytkowicz, Understanding the efficiency of social tagging systems using information theory, Proceedings of the nineteenth ACM conference on Hypertext and hypermedia, June 19-21, 2008, Pittsburgh, PA, USA
	Morgan Ames , Mor Naaman, Why we tag: motivations for annotation in mobile and online media, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA
	Rick Wash, Incentive design for home computer security, CHI '07 extended abstracts on Human factors in computing systems, April 28-May 03, 2007, San Jose, CA, USA
	Timo Volkmer , James A. Thom , S. M. M. Tahaghoghi, Exploring human judgement of digital imagery, Proceedings of the thirtieth Australasian conference on Computer science, p.151-160, January 30-February 02, 2007, Ballarat, Victoria, Australia
	Bageshree Shevade , Hari Sundaram , Lexing Xie, Modeling personal and social network context for event annotation in images, Proceedings of the 2007 conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada
	Sean Casey , Ben Kirman , Duncan Rowland, The gopher game: a social, mobile, locative game with user generated content and peer review, Proceedings of the international conference on Advances in computer entertainment technology, June 13-15, 2007, Salzburg, Austria
	Qi Su , Dmitry Pavlov , Jyh-Herng Chow , Wendell C. Baker, Internet-scale collection of human-reviewed data, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Shane Ahern , Simon King , Mor Naaman , Rahul Nair, Summarization of online image collections via implicit feedback, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Christian R. Prause , Stefan Apelt, An approach for continuous inspection of source code, Proceedings of the 6th international workshop on Software quality, May 10-10, 2008, Leipzig, Germany
	Michael J. Muller, Comparing tagging vocabularies among four enterprise tag-based services, Proceedings of the 2007 international ACM conference on Supporting group work, November 04-07, 2007, Sanibel Island, Florida, USA
	David Milne , David M. Nichols , Ian H. Witten, A competitive environment for exploratory query expansion, Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, June 16-20, 2008, Pittsburgh PA, PA, USA
	Tom Yeh , John J. Lee , Trevor Darrell, Photo-based question answering, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Jens Edlund , Joakim Gustafson , Mattias Heldner , Anna Hjalmarsson, Towards human-like spoken dialogue systems, Speech Communication, v.50 n.8-9, p.630-645, August, 2008
	Daniel Ramage , Paul Heymann , Christopher D. Manning , Hector Garcia-Molina, Clustering the tagged web, Proceedings of the Second ACM International Conference on Web Search and Data Mining, February 09-12, 2009, Barcelona, Spain
	Darius Pfitzner , Kenneth Treharne , David Powers, User keyword preference: the Nwords and Rwords experiments, International Journal of Internet Protocol Technology, v.3 n.3, p.149-158, October 2008
	Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008
	Rick Wash , Jeffrey K. MacKie-Mason, Security when people matter: structuring incentives for user behavior, Proceedings of the ninth international conference on Electronic commerce, August 19-22, 2007, Minneapolis, MN, USA
	Pradeep Shenoy , Desney S. Tan, Human-aided computing: utilizing implicit human processing to classify images, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Christian R. Prause , Markus Eisenhauer, Social aspects of a continuous inspection platform for software source code, Proceedings of the 2008 international workshop on Cooperative and human aspects of software engineering, p.85-88, May 13-13, 2008, Leipzig, Germany
	Casey Dugan , Michael Muller , David R. Millen , Werner Geyer , Beth Brownholtz , Marty Moore, The dogear game: a social bookmark recommender system, Proceedings of the 2007 international ACM conference on Supporting group work, November 04-07, 2007, Sanibel Island, Florida, USA
	Duarte Gonçalves , Rui Jesus , Nuno Correia, A gesture based game for image tagging, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy
	Luis von Ahn, Human computation, Proceedings of the 4th international conference on Knowledge capture, p.5-6, October 28-31, 2007, Whistler, BC, Canada
	Michael G. Christel, Establishing the utility of non-text search for news video retrieval with real world users, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany
	Amit Zunjarwad , Hari Sundaram , Lexing Xie, Contextual wisdom: social relations and correlations for multimedia event annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany
	Haakon Ringberg , Augustin Soule , Jennifer Rexford, WebClass: adding rigor to manual labeling of traffic anomalies, ACM SIGCOMM Computer Communication Review, v.38 n.1, January 2008
	Yang Yang , Bin B. Zhu , Rui Guo , Linjun Yang , Shipeng Li , Nenghai Yu, A comprehensive human computation framework: with application to image labeling, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Ramesh Jain, Multimedia information retrieval: watershed events, Proceeding of the 1st ACM international conference on Multimedia information retrieval, October 30-31, 2008, Vancouver, British Columbia, Canada
	Supheakmungkol Sarin , Toshinori Nagahashi , Tadashi Miyosawa , Wataru Kameyama, On the design and exploitation of user's personal and public information for semantic personal digital photograph annotation, Advances in Multimedia, v.2008 n.3, p.1-16, January 2008
	Xian-Sheng Hua , Guo-Jun Qi, Online multi-label active annotation: towards large-scale content-based video search, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Wenn-Chieh Tsai , Yuan-Hsiang Lee , Tsung-Hsiang Chang , Chien-Ju Ho , Jane Yung-jen Hsu, Designing human-computer multi-agent collaboration in productive multi-player games, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal
	Albert A. Angehrn , Katrina Maxwell , Bertrand Sereno, Enhancing social interaction in competence development networks: a conceptual framework, International Journal of Learning Technology, v.3 n.4, p.424-442, July 2008
	Noah Snavely , Steven M. Seitz , Richard Szeliski, Modeling the World from Internet Photo Collections, International Journal of Computer Vision, v.80 n.2, p.189-210, November 2008
	Luis von Ahn , Laura Dabbish, Designing games with a purpose, Communications of the ACM, v.51 n.8, August 2008
	Shawn R. Jeffery , Michael J. Franklin , Alon Y. Halevy, Pay-as-you-go user feedback for dataspace systems, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada
	Patrick Chiu , Jeffrey Huang , Maribeth Back , Nicholas Diakopoulos , John Doherty , Wolf Polak , Xiaohua Sun, mTable: browsing photos and videos on a tabletop system, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Allan Hanbury, A survey of methods for image annotation, Journal of Visual Languages and Computing, v.19 n.5, p.617-627, October, 2008
	Garrett Brown , Travis Howe , Micheal Ihbe , Atul Prakash , Kevin Borders, Social networks and context-aware spam, Proceedings of the ACM 2008 conference on Computer supported cooperative work, November 08-12, 2008, San Diego, CA, USA
	Zhiwei Li , Lei Zhang , Wei-Ying Ma, Delivering online advertisements inside images, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Gal Chechik , Eugene Ie , Martin Rehn , Samy Bengio , Dick Lyon, Large-scale content-based audio retrieval from text queries, Proceeding of the 1st ACM international conference on Multimedia information retrieval, October 30-31, 2008, Vancouver, British Columbia, Canada
	Nicholas Diakopoulos , Kurt Luther , Irfan Essa, Audio Puzzler: piecing together time-stamped speech transcripts with a puzzle game, Proceeding of the 16th ACM international conference on Multimedia, October 26-31, 2008, Vancouver, British Columbia, Canada
	Rodrigo F. Carvalho , Sam Chapman , Fabio Ciravegna, Attributing semantics to personal photographs, Multimedia Tools and Applications, v.42 n.1, p.73-96, March 2009
	Duarte Gonçalves , Rui Jesus , Filipe Grangeiro , Teresa Romão , Nuno Correia, Tag around: a 3D gesture game for image annotation, Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology, December 03-05, 2008, Yokohama, Japan
	Robert Speer , Jayant Krishnamurthy , Catherine Havasi , Dustin Smith , Henry Lieberman , Kenneth Arnold, An interface for targeted collection of common sense knowledge using a mixture model, Proceedings of the 13th international conference on Intelligent user interfaces, February 08-11, 2009, Sanibel Island, Florida, USA
	Narjes Sharif Razavian , Stephan Vogel, The web as a platform to build machine translation resources, Proceeding of the 2009 international workshop on Intercultural collaboration, February 20-21, 2009, Palo Alto, California, USA
	Kurt Alfred Kluever , Richard Zanibbi, Balancing usability and security in a video CAPTCHA, Proceedings of the 5th Symposium on Usable Privacy and Security, July 15-17, 2009, Mountain View, California
	Pei-Yun Hsueh , Prem Melville , Vikas Sindhwani, Data quality from crowdsourcing: a study of annotation selection criteria, Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, June 05-05, 2009, Boulder, Colorado
	Stephen Robertson , Milan Vojnovic , Ingmar Weber, Rethinking the ESP game, Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Rich Gossweiler , Maryam Kamvar , Shumeet Baluja, What's up CAPTCHA?: a CAPTCHA based on image orientation, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Yuki Arase , Xing Xie , Manni Duan , Takahiro Hara , Shojiro Nishio, A game based approach to assign geographical relevance to web images, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Paul N. Bennett , David Maxwell Chickering , Anton Mityagin, Learning consensus opinion: mining data from a labeling game, Proceedings of the 18th international conference on World wide web, April 20-24, 2009, Madrid, Spain
	Ziyu Guan , Jiajun Bu , Qiaozhu Mei , Chun Chen , Can Wang, Personalized tag recommendation using graph-based ranking on multi-type interrelated objects, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Hao Ma , Raman Chandrasekar , Chris Quirk , Abhishek Gupta, Page hunt: improving search engines using human computation games, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Catherine C. Marshall, No bull, no spin: a comparison of tags with other forms of user metadata, Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, June 15-19, 2009, Austin, TX, USA
	Rada Mihalcea , Chee Wee Leong, Toward communicating simple sentences using pictorial representations, Machine Translation, v.22 n.3, p.153-173, September 2008
	Ran Halprin , Moni Naor, Games for extracting randomness, Proceedings of the 5th Symposium on Usable Privacy and Security, July 15-17, 2009, Mountain View, California
	Gabe Johnson , Ellen Yi-Luen Do, Games for sketch data collection, Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, August 01-02, 2009, New Orleans, Louisiana
	Jeff Orkin , Deb Roy, Automatic learning and generation of social behavior from collective human gameplay, Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems, May 10-15, 2009, Budapest, Hungary
	Gabriella Kazai , Natasa Milic-Frayling , Jamie Costello, Towards methods for the collective gathering and quality control of relevance assessments, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Shaili Jain , Yiling Chen , David C. Parkes, Designing incentives for online question and answer forums, Proceedings of the tenth ACM conference on Electronic commerce, July 06-10, 2009, Stanford, California, USA
	Radu Jurca , Boi Faltings, Mechanisms for making crowds truthful, Journal of Artificial Intelligence Research, v.34 n.1, p.209-253, January 2009
	Edith Law , Luis von Ahn, Input-agreement: a new mechanism for collecting data using human computation games, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Severin Hacker , Luis von Ahn, Matchin: eliciting user preferences with an online game, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Marek Bell , Stuart Reeves , Barry Brown , Scott Sherwood , Donny MacMillan , John Ferguson , Matthew Chalmers, EyeSpy: supporting navigation through play, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Raphael Hoffmann , Saleema Amershi , Kayur Patel , Fei Wu , James Fogarty , Daniel S. Weld, Amplifying community content creation with mixed initiative information extraction, Proceedings of the 27th international conference on Human factors in computing systems, April 04-09, 2009, Boston, MA, USA
	Tsung-Hsiang Chang , Chien-Ju Ho , Jane Yung-Jen Hsu, The PhotoSlap game: play to annotate, Proceedings of the 22nd national conference on Artificial intelligence, p.1966-1967, July 22-26, 2007, Vancouver, British Columbia, Canada
	Chien-Ju Ho , Tsung-Hsiang Chang , Jane Yung-Jen Hsu, PhotoSlap: a multi-player online game for semantic annotation, Proceedings of the 22nd national conference on Artificial intelligence, p.1359-1364, July 22-26, 2007, Vancouver, British Columbia, Canada