User preference choices for complex question answering

ABSTRACT

Question answering systems increasingly need to deal with complex information needs that require more than simple factoid answers. The evaluation of such systems is usually carried out using precision- or recall-based system performance metrics. Previous work has demonstrated that when users are shown two search result lists side-by-side, they can reliably differentiate between the qualities of the lists. We investigate the consistency between this user-based approach and system-oriented metrics in the question answering environment. Our initial results indicate that the two methodologies show a high level of disagreement.

REFERENCES

• 1: H. T. Dang and D. Kelly and J. Lin. Overview of the TREC-2007 Question Answering Track. In TREC 2007 Notebook, November 2007, Gaithersburg, Maryland.
• 2: Jimmy Lin , Dina Demner-Fushman, Will pyramids built of nuggets topple over?, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.383-390, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220884]
• 3: Ian Ruthven , Leif Azzopardi Glasgow , Mark Baillie , Ralf Bierig , Emma Nicol , Simon Sweeney , Murat Yakici, Intra-assessor consistency in question answering, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277879]
• 4: Paul Thomas , David Hawking, Evaluation by comparing result sets in context, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183632]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Human Factors, Performance

Keywords:
TREC, ciQA, human preference and judgement