Collecting paraphrase corpora from volunteer contributors

ABSTRACT

Extensive and deep paraphrase corpora are important for a variety of natural language processing and user interaction tasks. In this paper, we present an approach which i) collects multiple paraphrases per given item from volunteers and ii) incentivises responsible contributions by volunteer contributors. Our approach is to solicit paraphrases from Web volunteers, both collecting new paraphrases with no prompting and asking contributors to guess partially obfuscated paraphrases. To test the approach, we have implemented an online game, 1001 Paraphrases (http://ai-games.org/paraphrase.html), and deployed it to collect 20,944 entries focused on paraphrases of 400 statements. The approach complements existing text extraction methods and has some inherent unique advantages. We present and motivate our design as well as share preliminary observations and lessons learned about the performance of the approach.

REFERENCES

• 1: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 2: Toward conversational human-computer interaction, AI Magazine, v.22 n.4, p.27-37, Winter 2001
• 3: Regina Barzilay , Lillian Lee, Learning to paraphrase: an unsupervised approach using multiple-sequence alignment, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.16-23, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073448]
• 4: Belasco, A., Curtis, J., Kahlert, R., Klein, C., Mayans, C., Reagan, P. 2002. Representing Knowledge Gaps Effectively. In Practical Aspects of Knowledge Management, (PAKM), Vienna, Austria, December 2-3.
• 5: Timothy Chklovski , Rada Mihalcea, Building a sense tagged corpus with open mind word expert, Proceedings of the ACL-02 workshop on Word sense disambiguation: recent successes and future directions, p.116-122, July 11, 2002 [doi>10.3115/1118675.1118692]
• 6: Chklovski, T. 2003a. Using Analogy to Acquire Commonsense Knowledge from Human Contributors, PhD thesis. MIT Artificial Intelligence Laboratory technical report AITR-2003-002.
• 7: Timothy Chklovski, Learner: a system for acquiring commonsense knowledge by analogy, Proceedings of the 2nd international conference on Knowledge capture, October 23-25, 2003, Sanibel Island, FL, USA [doi>10.1145/945645.945650]
• 8: Chklovski, T. and Gil, Y. 2005. An Analysis of Knowledge Collected from Volunteer Contributors. In Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI-05).
• 9: Timothy Chklovski, Designing interfaces for guided collection of knowledge about everyday objects from volunteers, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040910]
• 10: Dolan, W. B., Quirk, C., and Brockett, C. 2004. Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources. In Proceedings of COLING 2004, Geneva, Switzerland.
• 11: Gupta, R., and Kochenderfer, M. 2004. Common sense data acquisition for indoor mobile robots. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04).
• 12: Kevin Knight , Daniel Marcu, Statistics-Based Summarization - Step One: Sentence Compression, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.703-710, July 30-August 03, 2000
• 13: Mihalcea, R., and Chklovski, T. 2004. Building Sense Tagged Corpora with Volunteer Contributions over the Web. In Current Issues in Linguistic Theory: Recent Advances in Natural Language Processing, Nicolas Nicolov and Ruslan Mitkov (eds), John Benjamins Publishers.
• 14: Narayanan, S., Ananthakrishnan, S., Belvin, R., Ettelaie, E. Ganjavi, S. Georgiou, P., Hein, C., Kadambe, S., Knight, K., Marcu, D., Neely, H., Srinivasamurthy, N., Traum, D. and Wang, D. 2003. Transonics: A Speech to Speech System for English-Persian Interactions. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (IEEE ASRU).
• 15: Stork, D. 2003. Invited talk at the Workshop on Distributed and Collaborative Knowledge Capture (DC-KCAP), held in conjunction with the International conference on Knowledge Capture (K-CAP 2003).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Knowledge acquisition

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Design

Keywords:
interfaces for knowledge elicitation, paraphrase corpora, volunteer contributor-based knowledge acquisition