Building a question answering test collection

ABSTRACT

The TREC-8 Question Answering (QA) Track was the first large-scale evaluation of domain-independent question answering systems. In addition to fostering research on the QA task, the track was used to investigate whether the evaluation methodology used for document retrieval is appropriate for a different natural language processing task. As with document relevance judging, assessors had legitimate differences of opinions as to whether a response actually answers a question, but comparative evaluation of QA systems was stable despite these differences. Creating a reusable QA test collection is fundamentally more difficult than creating a document retrieval test collection since the QA task has no equivalent to document identifiers.

REFERENCES

• 1: Douglas E. Appelt , Jerry R. Hobbs , John Bear , David Israel , Megumi Kameyama , David Martin , Karen Myers , Mabry Tyson, SRI International FASTUS system: MUC-6 test results and analysis, Proceedings of the 6th conference on Message understanding, November 06-08, 1995, Columbia, Maryland [doi>10.3115/1072399.1072420]
• 2: R. Weischedel, BEN: description of the PLUM system as used for MUC-6, Proceedings of the 6th conference on Message understanding, November 06-08, 1995, Columbia, Maryland [doi>10.3115/1072399.1072407]
• 3: Eric Breck, John Burger, Lisa Ferro, David House, Marc Light, and Indeueet Mani. A sys called Qanda. In Proceedings of the Eighth Text REtrieval Conference (TREC- 8), pages 4.43-451, November 1999. Notebook draft.
• 4: Robin D. Burke , Kristian J. Hammond , Vladimir A. Kulyukin , Steven L. Lytinen , N. Tomuro , S. Schoenberg, Question Answering from Frequently Asked Question Files: Experiences with the FAQ Finder System, University of Chicago, Chicago, IL, 1997
• 5: Paul Cohen, Robert Schrag, Eric Jones, Adam Pease, Albert Lin, Barbara Starr, David Gunning, and Murray Burke. The DARPA high-performance knowledge bases project. AI Magazine, pages 25-49, Winter 1998.
• 6: Laura L. Downey , Dawn M. Tice, A usability case study using TREC and ZPRISE, Information Processing and Management: an International Journal, v.35 n.5, p.589-603, Sept. 1999 [doi>10.1016/S0306-4573(99)00013-8]
• 7: Boris Katz. From sentence processing to information access on the world wide web. Paper presented at the AAAI Spnng Symposium on Natural Language Processing for the World Wide Web, 1997. Electronic version at http ://www.ai.ait.edu/ people/boris/webaccess.
• 8: Julian Kupiec, MURAX: a robust linguistic approach for question answering using an on-line encyclopedia, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.181-190, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160717]
• 9: M.E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval. 4:343-359, 1969.
• 10: Joel Martin and Chris Lankester. Ask Me Tomorrow: The University of Ottawa question answering system. In Proceedings of the Eighth Text REtrieval Conference (TREC- 8), pages 575-583, November 1999. Notebook draft.
• 11: John O'Connor. Answer-passage retrieval by text searching. Journal of the American Society for Information Science, pages 227-239, July 1980.
• 12: Linda Schamber. Relevance and information behavior. Annual Review of in formation Science and Technology, 29:3- 48, 1994.
• 13: Alan Stuart. Kendalr s tau. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, volume 4, pages 367-369. John Wiley & Sons, 1983.
• 14: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 15: Ellen M. Voorhees , Donna Harman, Overview of the sixth text REtrieval conference (TREC-6), Information Processing and Management: an International Journal, v.36 n.1, p.3-35, Jan. 2000 [doi>10.1016/S0306-4573(99)00043-6]
• 16: Ellen M. Voorhees and Dawn M. Tice. The TREC-8 question answering track evaluation. In E.M. Voorhees and D.K. Harman, editors, Proceedings of the Eighth Text RE- trieval Conference (TREC-8 ). Electronic version available at http://trec.nist.gov/pubs.htrul, 2000.
• 17: B. Webber. Question answering. In Stuart C. Shapiro, editor, Encyclopedia of Articficial Intelligence, volume 2, pages 814.-822. Wiley, 1987.
• 18: Terry Winograd. Five lectures on artificial intelligence. In A. Zampolli, editor, Lingusitic Structures Processing, volume 5 of Fundamental Studies in Computer Science, pages 399-520. North Holland, 1977.
• 19: W. A. Woods. Lunar rocks in natural english: Explorations in natural language question answering. In A. Zampolli, editor, Lingusitic Structures Processing, volume 5 of Fundamental Studies in Computer Science, pages 521-569. North Holland, 1977.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION

General Terms:
Design, Documentation