A demonstration of human computation using the Phrase Detectives annotation game

ABSTRACT

The goal of the ANAWIKI project is to experiment with Web collaboration and human computation to create largescale linguistically annotated corpora. We will present ongoing work and initial results of Phrase Detectives, a game designed to collect judgments about anaphoric annotations.

REFERENCES

• 1: J. Chamberlain, M. Poesio, and U. Kruschwitz. Phrase Detectives: A Web-based Collaborative Annotation Game. In Proceedings of the International Conference on Semantic Systems (I-Semantics'08), Graz, 2008.
• 2: J. Chamberlain, M. Poesio, and U. Kruschwitz. A new life for a dead parrot: Incentive structures in the Phrase Detectives game. In Proceedings of the Webcentives Workshop at WWW'09, Madrid, 2009.
• 3: J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37--46, 1960.
• 4: N. L. Johnson, S. Rasmussen, C. Joslyn, L. Rocha, S. Smith, and M. Kantor. Symbiotic Intelligence: Self-Organizing Knowledge on Distributed Networks Driven by Human Interaction. In Proceedings of the Sixth International Conference on Artificial Life. MIT Press, 1998.
• 5: U. Kruschwitz, J. Chamberlain, and M. Poesio. (Linguistic) Science Through Web Collaboration in the ANAWIKI Project. In Proceedings of WebSci'09, Athens, 2009.
• 6: J. Mrozinski, E. Whittaker, and S. Furui. Collecting a why-question corpus for development and evaluation of an automatic QA-system. In Proceedings of ACL-08: HLT, pages 443--451, Columbus, Ohio, June 2008. Association for Computational Linguistics.
• 7: M. Poesio. Discourse annotation and semantic annotation in the gnome corpus. In Proceedings of the ACL Workshop on Discourse Annotation, 2004.
• 8: K. Siorpaes and M. Hepp. Games with a purpose for the semantic web. IEEE Intelligent Systems, 23(3):50--60, 2008.
• 9: J. Surowiecki. The Wisdom of Crowds. Anchor, 2005.
• 10: L. von Ahn. Games with a purpose. Computer, 39(6):92--94, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Keywords:
anaphoric annotation, human computation, natural language processing, web collaboration, web games