The effect of topic set size on retrieval experiment error

ABSTRACT

Retrieval mechanisms are frequently compared by computing the respective average scores for some effectiveness metric across a common set of information needs or topics, with researchers concluding one method is superior based on those averages. Since comparative retrieval system behavior is known to be highly variable across topics, good experimental design requires that a "sufficient" number of topics be used in the test. This paper uses TREC results to empirically derive error rates based on the number of topics used in a test and the observed difference in the average scores. The error rates quantify the likelihood that a different set of topics of the same size would lead to a different conclusion. We directly compute error rates for topic sets up to size 25, and extrapolate those rates for larger topic set sizes. The error rates found are larger than anticipated, indicating researchers need to take care when concluding one method is better than another, especially if few topics are used.

REFERENCES

• 1: David Banks , Paul Over , Nien-Fan Zhang, Blind Men and Elephants: Six Approaches to TREC data, Information Retrieval, v.1 n.1-2, p.7-34, 1999 [doi>10.1023/A:1009984519381]
• 2: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 3: C. W. Cleverdon, J. Mills, and E. M. Keen. Factors determining the performance of indexing systems. Two volumes, Cranfield, England, 1968.
• 4: E. Michael Keen, Presenting results of experimental retrieval comparisons, Information Processing and Management: an International Journal, v.28 n.4, p.491-502, July-Aug. 1992 [doi>10.1016/0306-4573(92)90006-L]
• 5: M.-D. Lacasse. FUDGIT A multi-purpose data-processing and fitting program user's manual version 2.31. Technical report, Center for the Physics of Materials and Department of Physics, Montreal, Canada, April 1993.
• 6: Gerard Salton, The state of retrieval system evaluation, Information Processing and Management: an International Journal, v.28 n.4, p.441-449, July-Aug. 1992 [doi>10.1016/0306-4573(92)90002-H]
• 7: K. Sparck Jones. Automatic indexing. Journal of Documentation, 30:393--432, 1974.
• 8: K. Sparck Jones and C. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59--75, 1976.
• 9: M. Twain. Life on the Mississippi, 1883. Reprinted at http://www.lhup.edu/~dsimanek/twain.htm.
• 10: Ellen M. Voorhees , Donna Harman, Overview of the sixth text REtrieval conference (TREC-6), Information Processing and Management: an International Journal, v.36 n.1, p.3-35, Jan. 2000 [doi>10.1016/S0306-4573(99)00043-6]
• 11: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
measurement error, test collections