Automatic evaluation of world wide web search services

ABSTRACT

Users of the World-Wide Web are not only confronted by an immense overabundance of information, but also by a plethora of tools for searching for the web pages that suit their information needs. Web search engines differ widely in interface, features, coverage of the web, ranking methods, delivery of advertising, and more. In this paper, we present a method for comparing search engines automatically based on how they rank known item search results. Because the engines perform their search on overlapping (but different) subsets of the web collected at different points in time, evaluation of search engines poses significant challenges to the traditional information retrieval methodology. Our method uses known item searching; comparing the relative ranks of the items in the search engines' rankings. Our approach automatically constructs known item queries using query log analysis and automatically constructs the result via analysis of editor comments from the ODP (Open Directory Project). Additionally, we present our comparison on five (Lycos, Netscape, Fast, Google, HotBot) well-known search services and find that some services perform known item searches better than others, but the majority are statistically equivalent.

REFERENCES

• 1: David Hawking , Nick Craswell , Paul Thistlewaite , Donna Harman, Results and challenges in Web search evaluation, Proceeding of the eighth international conference on World Wide Web, p.1321-1330, May 1999, Toronto, Canada
• 2: Amit Singhal , Marcin Kaszkiel, A case study in web search using TREC algorithms, Proceedings of the 10th international conference on World Wide Web, p.708-716, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372186]
• 3: Michael Gordon , Praveen Pathak, Finding information on the World Wide Web: the retrieval effectiveness of search engines, Information Processing and Management: an International Journal, v.35 n.2, p.141-180, March 1999 [doi>10.1016/S0306-4573(98)00041-7]
• 4: H. Vernon Leighton , Jaideep Srivastava, First 20 precision among World Wide Web search services (search engines), Journal of the American Society for Information Science, v.50 n.10, p.870-881, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:10<870::AID-ASI4>3.3.CO;2-7]
• 5: D. Hawking, TREC Web Track Guidelines, 2001, http://www.ted.cmis.csiro.au/TRECWeb/guidelines_2001.html
• 6: Chris Buckley, TREC Web Track mailing list, 2001.
• 7: Open Directory Project, http://www.dmoz.org
• 8: Amanda Spink , Bernard J. Jansen , Dietmar Wolfram , Tefko Saracevic, From E-Sex to E-Commerce: Web Search Changes, Computer, v.35 n.3, p.107-109, March 2002 [doi>10.1109/2.989940]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Experimentation, Performance

Keywords:
IR evaluation, automatic mean reciprocal ranking