Can people collaborate to improve the relevance of search results?

ABSTRACT

Search engines are among the most-used resources on the internet. However, even today's most successful search engines struggle to provide high quality search results. According to recent studies as many as 50 percent of web search sessions fail to find any relevant results for the searcher. Researchers have proposed social search techniques, in which early searchers provide feedback that is used to improve relevance for later searchers. In this paper we investigate foundational questions of social search. In particular, we directly assess the degree of agreement among users about the relevance ranking of search results. We developed a simulated search engine interface that systematically randomizes Google's normal relevance ordering of the items presented to users. Our results show that (a) people are biased toward items in the top of the search lists, even if the list is randomized; (b) people explicit feedback is not biased and (c) people's shared preferences do not always agree with Google's result order. These results suggest that social search techniques might improve the effectiveness of web search engines.

REFERENCES

• 1: Pass, G., Chowdhury, A., Torgeson, C. 2006. Picture of Search. In Proceedings of InfoScale 2006
• 2: Ricardo A. Baeza-Yates , Felipe Saint-Jean , Carlos Castillo, Web Structure, Dynamics and Page Quality, Proceedings of the 9th International Symposium on String Processing and Information Retrieval, p.117-130, September 11-13, 2002
• 3: Junghoo Cho , Sourashis Roy, Impact of search engines on page popularity, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988676]
• 4: Junghoo Cho , Sourashis Roy , Robert E. Adams, Page quality: in search of an unbiased web ranking, Proceedings of the 2005 ACM SIGMOD international conference on Management of data, June 14-16, 2005, Baltimore, Maryland [doi>10.1145/1066157.1066220]
• 5: Sandeep Pandey , Sourashis Roy , Christopher Olston , Junghoo Cho , Soumen Chakrabarti, Shuffling a stacked deck: the case for partially randomized ranking of search engine results, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway
• 6: Mark T. Keane , Maeve O'Brien , Barry Smyth, Are people biased in their use of search engines?, Communications of the ACM, v.51 n.2, p.49-52, February 2008 [doi>10.1145/1314215.1314224]
• 7: Amanda Spink , Dietmar Wolfram , Major B. J. Jansen , Tefko Saracevic, Searching the Web: the public and their queries, Journal of the American Society for Information Science and Technology, v.52 n.3, p.226-234, Feb. 1, 2001 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1591>3.3.CO;2-I]
• 8: Saleema Amershi , Meredith Ringel Morris, CoSearch: a system for co-located collaborative web search, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357311]
• 9: Dan Morris , Meredith Ringel Morris , Gina Venolia, SearchBar: a search-centric web history for task resumption and information re-finding, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357242]
• 10: Barry Smyth, A Community-Based Approach to Personalizing Web Search, Computer, v.40 n.8, p.42-50, August 2007 [doi>10.1109/MC.2007.259]
• 11: Jaime Teevan , Eytan Adar , Rosie Jones , Michael A. S. Potts, Information re-retrieval: repeat queries in Yahoo's logs, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277770]
• 12: Anne Aula , Natalie Jhaveri , Mika Käki, Information search and re-access strategies of experienced web users, Proceedings of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1060745.1060831]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Selection process; Relevance feedback
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Evaluation/methodology; Computer-supported cooperative work; Collaborative computing

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
browsing, explicit feedback, recommender systems, social access patterns, social search