Modelling epistemic uncertainty in ir evaluation

ABSTRACT

Modern information retrieval (IR) test collections violate the completeness assumption of the Cranfield paradigm. In order to maximise the available resources, only a sample of documents (i.e. the pool) are judged for relevance by a human assessor(s). The subsequent evaluation protocol does not make any distinctions between assessed or unassesseddocuments, as documents that are not in the pool are assumedto be not relevant for the topic. This is beneficial from a practical point of view, as the relative performance can be compared with confidence if the experimental conditions are fair for all systems. However, given the incompleteness of relevance assessments, two forms of uncertainty emerge during evaluation. The first is Aleatory uncertainty, which refers to variation in system performance across the topic set, which is often addressed through the use of statistical significance tests. The second form of uncertainty is Epistemic, which refers to the amount of knowledge (or ignorance) we have about the estimate of a system's performance. Epistemic uncertainty is a consequence of incompleteness and is not addressed by the current evaluation protocol. In this study, we present a first attempt at modelling both aleatory and epistemic uncertainty associatedwith IR evaluation. We aim to account for both the variability associated with system performance and the amount of knowledge known about the performance estimate.

REFERENCES

• 1: Mark Baillie , Leif Azzopardi , Ian Ruthven, Evaluating epistemic uncertainty under incomplete assessments, Information Processing and Management: an International Journal, v.44 n.2, p.811-837, March, 2008 [doi>10.1016/j.ipm.2007.04.002]
• 2: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148284]
• 3: G. Shafer. A Mathematical Theory of Evidence. Princeton University Press, 1976.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Measurement, Reliability

Keywords:
metrics, performance evaluation, uncertainty