Relevance criteria for e-commerce: a crowdsourcing-based experimental analysis

ABSTRACT

We discuss the concept of relevance criteria in the context of e-Commerce search. A vast body of research literature describes the beyond-topical criteria used to determine the relevance of the document to the need. We argue that in an e-Commerce scenario there are some differences, and novel and different criteria can be used to determine relevance. We experimentally validate this hypothesis by means of Amazon Mechanical Turk using a crowdsourcing approach.

REFERENCES

• 1: Omar Alonso , Daniel E. Rose , Benjamin Stewart, Crowdsourcing for relevance evaluation, ACM SIGIR Forum, v.42 n.2, December 2008 [doi>10.1145/1480506.1480508]
• 2: Carol L. Barry , Linda Schamber, Users' criteria for relevance evaluation: a cross-situational comparison, Information Processing and Management: an International Journal, v.34 n.2-3, p.219-236, 1998 [doi>10.1016/S0306-4573(97)00078-2]
• 3: Aniket Kittur , Ed H. Chi , Bongwon Suh, Crowdsourcing user studies with Mechanical Turk, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357127]
• 4: Stefano Mizzaro, Relevance: the whole history, Journal of the American Society for Information Science, v.48 n.9, p.810-832, Sept. 1997 [doi>10.1002/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U]
• 5: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.1915-1933, November 2007 [doi>10.1002/asi.v58:13]
• 6: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.2126-2144, November 2007 [doi>10.1002/asi.v58:13]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Performance

Keywords:
IR evaluation, relevance, relevance criteria, user study