Query side evaluation: an empirical analysis of effectiveness and effort

ABSTRACT

Typically, Information Retrieval evaluation focuses on measuring the performance of the system's ability at retrieving relevant information, and not the query's ability. However, the effectiveness of a retrieval system is strongly influenced by the quality of the query submitted. In this paper, the effectiveness and effort of querying is empirically examined in the context of the Principle of Least Effort, Zipf's Law and the Law of Diminishing Returns. This query focused investigation leads to a number of novel findings which should prove useful in the development of future retrieval methods and evaluation techniques. While, also motivating further research into query side evaluation.

REFERENCES

• 1: G.K. Zipf. Human Behavior and the Principle of Least-Effort. Addison-Wesley, 1949.
• 2: H. S. Heaps, Information Retrieval: Computational and Theoretical Aspects, Academic Press, Inc., Orlando, FL, 1978
• 3: C.R.S.A. Clauset and M.E.J. Newman. Power-law distributions in empirical data. URL http://arxiv.org/abs/0706.1062v1, 2007.
• 4: Avi Arampatzis , Jaap Kamps, A study of query length, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390517]
• 5: Leif Azzopardi , Maarten de Rijke , Krisztian Balog, Building simulated queries for known-item topics: an analysis using six european languages, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277820]
• 6: R.K. Belew. Finding Out About. Cambridge Univ. Press, 2000.
• 7: David C. Blair, The challenge of commercial document retrieval, part I: major issues, and a framework based on search exhaustivity, determinacy of representation and document collection size, Information Processing and Management: an International Journal, v.38 n.2, p.273-291, March 2002 [doi>10.1016/S0306-4573(01)00024-3]
• 8: S. C. Bradford, “Sources of information on specific subjects”, Journal of Information Science, v.10 n.4, p.173-180, 1985 [doi>10.1177/016555158501000406]
• 9: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148284]
• 10: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 11: Masashi Inoue , Naonori Ueda, Retrieving lightly annotated images using image similarities, Proceedings of the 2005 ACM symposium on Applied computing, March 13-17, 2005, Santa Fe, New Mexico [doi>10.1145/1066677.1066914]
• 12: Chris Jordan , Carolyn Watters , Qigang Gao, Using controlled query generation to evaluate blind relevance feedback algorithms, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141818]
• 13: Diane Kelly , Vijay Deepak Dollu , Xin Fu, The loquacious user: a document-independent source of terms for query expansion, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076112]
• 14: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 15: A.J. Lotka. The frequency distribution of scientific productivity. Journal of the Washington Academy of Sciences, 16(12):317--324, 1926.
• 16: David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
• 17: Falk Scholer , Hugh E. Williams , Andrew Turpin, Query association surrogates for Web search: Research Articles, Journal of the American Society for Information Science and Technology, v.55 n.7, p.637-650, May 2004 [doi>10.1002/asi.v55:7]
• 18: Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076121]
• 19: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]
• 20: Y. Zhao, F. Scholer, and Y. Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In ECIR 08: Proceedings of the European Conference in Information Retrieval, pages 52--64, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Experimentation, Theory

Keywords:
evaluation, simulation