Exploring evaluation metrics: GMAP versus MAP

ABSTRACT

In retrieval experiments, an effectiveness metrics is used to generate a score for each system-topic pair being tested. It is then usual to average the system-topic scores to obtain a system score, which is used for the purpose of system comparison. In this paper we explore the ramifications of using the geometric mean (GMAP), rather than the arithmetic mean (MAP) when computing an aggregate system score from a set of system-topic scores. We find that GMAP does indeed handle variability in topic difficulty more consistently than does the usual MAP aggregation method.

REFERENCES

• 1: S.Mizzaro. The good, the bad, the difficult, and the easy: Something wrong with information retrieval evaluation? In Proc. 30th European Conf. on Information Retrieval, Glasgow, Scotland, March-April 2008.
• 2: Stephen Robertson, On GMAP: and other transformations, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183630]
• 3: E. M. Voorhees. Overview of the TREC 2003 Robust Retrieval Track. In Proc. 12th Text REtrieval Conference (TREC 2003), Gaithersburg, Maryland, November 2003.
• 4: William Webber , Alistair Moffat , Justin Zobel, Score standardization for inter-collection comparison of retrieval systems, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390346]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Performance

Keywords:
average precision, evaluation, gmap, map