Evaluation measures for preference judgments

ABSTRACT

There has been recent interest in collecting user or assessor preferences, rather than absolute judgments of relevance, for the evaluation or learning of ranking algorithms. Since measures like precision, recall, and DCG are defined over absolute judgments, evaluation over preferences will require new evaluation measures that explicitly model them. We describe a class of such measures and compare absolute and preference measures over a large TREC collection.

REFERENCES

• 1: B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there: Preference judgments for relevance. In Proceedings of ECIR, pages 16--27, 2008.
• 2: M. E. Rorvig. The simple scalability of documents. JASIS, 41(8):590--598, 1990.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Measurement, Performance

Keywords:
evaluation, preference judgments