Crowdsourcing for relevance evaluation

ABSTRACT

Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

REFERENCES

• 1: Amazon Mechanical Turk, http://www.mturk.com
• 2: Jeff Barr , Luis Felipe Cabrera, AI gets a brain, Queue, v.4 n.4, May 2006 [doi>10.1145/1142055.1142067]
• 3: Brendan O'Connor, "Search Engine Relevance: An Empirical Test", http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/#more-35, accessed April 13, 2008.
• 4: Jeff Howe. "The Rise of Crowdsourcing". Wired, June 2006. http://www.wired.com/wired/archive/14.06/crowds.html
• 5: Peter Ingwersen , Kalervo Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 6: Thorsten Joachims , Filip Radlinski, Search Engines that Learn from Implicit Feedback, Computer, v.40 n.8, p.34-40, August 2007 [doi>10.1109/MC.2007.289]
• 7: Daniel E. Rose, "Why Is Web Search So Hard... to Evaluate?" Journal of Web Engineering, Vol. 3, Nos. 3 & 4, pp. 171--181, December 2004.
• 8: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.2126-2144, November 2007 [doi>10.1002/asi.v58:13]
• 9: Ellen M. Voorhees, TREC: Continuing information retrieval's tradition of experimentation, Communications of the ACM, v.50 n.11, November 2007 [doi>10.1145/1297797.1297822]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection; Classifier design and evaluation

General Terms:
Algorithms, Design, Performance