A picture of search

ABSTRACT

We survey many of the measures used to describe and evaluate the efficiency and effectiveness of large-scale search services. These measures, herein visualized versus verbalized, reveal a domain rich in complexity and scale. We cover six principle facets of search: the query space, users' query sessions, user behavior, operational requirements, the content space, and user demographics. While this paper focuses on measures, the measurements themselves raise questions and suggest avenues of further investigation.

REFERENCES

• 1: Steven M. Beitzel , Eric C. Jensen , Ophir Frieder , David D. Lewis , Abdur Chowdhury , Aleksander Kolcz, Improving Automatic Query Classification via Semi-Supervised Learning, Proceedings of the Fifth IEEE International Conference on Data Mining, p.42-49, November 27-30, 2005 [doi>10.1109/ICDM.2005.80]
• 2: A. Chowdhury, "Automatic Evaluation of Web Search Services," in Advances in Computers, vol. ISBN: 0-12-012164-6, 2005.
• 3: W. Xi, K. Sidhu, and A. Chowdhury, "Latent Query Stability," presented at International Conference on Information and Knowledge Engineering IKE, 2003.
• 4: Steven M. Beitzel , Eric C. Jensen , Abdur Chowdhury , David Grossman , Ophir Frieder, Hourly analysis of a very large topically categorized web query log, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009048]
• 5: S. Beitzel, E. Jensen, A. Chowdhury, D. Grossman, and O. Frieder, "Hourly Analysis of a Very Large Topically Categorized Web Query Log," JASIST, vol. (to appear), 2006.
• 6: N. Belkin, R. Oddy, and H. Brooks, "ASK for information retrieval. Part I: Background and theory; Part II: Results of a design study," Journal of Documentation, vol. 38, pp. 61--71, 145--164, 1982.
• 7: G. Murray, J. Lin, and A. Chowdhury, "Characterizing Web Search User "Sessions" with Hierarchical Agglomerative Clustering," presented at under review, 2006.
• 8: Daqing He , Ayse Göker , David J. Harper, Combining evidence for automatic web session identification, Information Processing and Management: an International Journal, v.38 n.5, p.727-742, September 2002 [doi>10.1016/S0306-4573(01)00060-7]
• 9: Andrei Broder, A taxonomy of web search, ACM SIGIR Forum, v.36 n.2, Fall 2002 [doi>10.1145/792550.792552]
• 10: Daniel E. Rose , Danny Levinson, Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988675]
• 11: Eric C. Jensen , Steven M. Beitzel , Ophir Frieder , Abdur Chowdhury, A framework for determining necessary query set sizes to evaluate web search effectiveness, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062926]
• 12: Steven M. Beitzel , Eric C. Jensen , Ophir Frieder , Abdur Chowdhury , Greg Pass, Surrogate scoring for improved metasearch precision, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076139]
• 13: Abdur Chowdhury , Greg Pass, Operational requirements for scalable search systems, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956945]
• 14: A. Gulli , A. Signorini, The indexable web is more than 11.5 billion pages, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062789]
• 15: P. Boutin, "Brewster Kahle made a copy of the Internet, Now, he wants your files," vol. 2005: Slate, 2005.
• 16: Luis Gravano , Vasileios Hatzivassiloglou , Richard Lichtenstein, Categorizing web queries according to geographical locality, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956925]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Distributed systems

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process
      H.3.4 Systems and Software
          Subjects: User profiles and alert services
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Algorithms, Design, Human Factors

Keywords:
distributed database searching, search methods, system modeling, user interfaces, user modeling