Internet-scale collection of human-reviewed data

ABSTRACT

Enterprise and web data processing and content aggregation systems often require extensive use of human-reviewed data (e.g. for training and monitoring machine learning-based applications). Today these needs are often met by in-house efforts or out-sourced offshore contracting. Emerging applications attempt to provide automated collection of human-reviewed data at Internet-scale. We conduct extensive experiments to study the effectiveness of one such application. We also study the feasibility of using Yahoo! Answers, a general question-answering forum, for human-reviewed data collection.

REFERENCES

• 1: Amazon Mechanical Turk. http://www.mturk.com/.
• 2: J. Angwin. On the offensive -- a problem for hot web outfits: Keeping pages free from porn. Wall Street Journal, May 2006.
• 3: S. Argamon-Engelson and I. Dagan. Committee-based sample selection for probabilistic classiers. Journal of Artificial Intelligence Research, 1999.
• 4: O. Benjelloun, H. Garcia-Molina, H. Kawai, T. Larson, D. Menestrina, Q. Su, S. Thavisomboon, and J. Widom. Generic Entity Resolution in the SERF Project. IEEE Data Engineering Bulletin, June 2006.
• 5: J. Breese, D. Heckerman, and C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In Proc. of Uncertainty in Artificial Intelligence, 1998.
• 6: David Cohn , Les Atlas , Richard Ladner, Improving Generalization with Active Learning, Machine Learning, v.15 n.2, p.201-221, May 1994 [doi>10.1023/A:1022673506211]
• 7: Helena Galhardas , Daniela Florescu , Dennis Shasha , Eric Simon , Cristian-Augustin Saita, Declarative Data Cleaning: Language, Model, and Algorithms, Proceedings of the 27th International Conference on Very Large Data Bases, p.371-380, September 11-14, 2001
• 8: Craig Gentry , Zulfikar Ramzan , Stuart Stubblebine, Secure distributed human computation, Proceedings of the 6th ACM conference on Electronic commerce, p.155-164, June 05-08, 2005, Vancouver, BC, Canada [doi>10.1145/1064009.1064026]
• 9: Google Image Labeler. http://images.google.com/imagelabeler/.
• 10: J. Hipp, U. Guntzer, and U. Grimmer. Data quality mining -- making a virtue of necessity. In Proc. of SIGMOD DMKD Workshop, 2001.
• 11: J. Howe. The rise of crowdsourcing. Wired, June 2006.
• 12: A. Koblin. The sheep market: Two cents worth. Master's thesis, UCLA, 2006.
• 13: Andrew McCallum , Kamal Nigam, Employing EM and Pool-Based Active Learning for Text Classification, Proceedings of the Fifteenth International Conference on Machine Learning, p.350-358, July 24-27, 1998
• 14: E. Rahm and H. Do. Data Cleaning: Problems and Current Approaches. IEEE Data Engineering Bulletin, December 2000.
• 15: Paul Resnick , Hal R. Varian, Recommender systems, Communications of the ACM, v.40 n.3, p.56-58, March 1997 [doi>10.1145/245108.245121]
• 16: Sunita Sarawagi , Anuradha Bhamidipaty, Interactive deduplication using active learning, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775087]
• 17: James Surowiecki, The Wisdom of Crowds, Anchor, 2005
• 18: Tenacious Search. http://openphi.net/tenacious/.
• 19: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]
• 20: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 21: Luis von Ahn , Ruoran Liu , Manuel Blum, Peekaboom: a game for locating objects in images, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124782]
• 22: Luis von Ahn , Mihir Kedia , Manuel Blum, Verbosity: a game for collecting common-sense facts, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124784]
• 23: L. von Ahn et al. The ESP Game. http://www.espgame.org/.
• 24: Yahoo! Answers.http://answers.yahoo.com/.
• 25: Yahoo! Suggestion Board. http://suggestions.yahoo.com/.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

Additional Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
data collection, human data, manual review