Overview of the TREC 2006 ciQA task

ABSTRACT

Growing interest in interactive systems for answering complex questions lead to the development of the complex, interactive QA (ciQA) task, introduced for the first time at TREC 2006. This paper describes the rationale and design of the ciQA task and the evaluation results. Thirty complex relationship questions based on five question templates were investigated using the AQUAINT collection of newswire text. Interaction forms were the primary vehicle for defining and capturing user-system interactions. In total, six groups participated in the ciQA task and contributed ten different sets of interaction forms. There were two main findings: baseline IR techniques are competitive for complex QA and interaction, at least as defined and implemented in this evaluation, did not appear to improve performance by much.

REFERENCES

• 1: J. Allan. 2005. HARD track overview in TREC 2005: High accuracy retrieval from documents. In TREC 2005.
• 2: H. Dang, J. Lin, and D. Kelly. 2006. Overview of the TREC 2006 question answering track. In TREC 2006.
• 3: S. Dumais and N. Belkin. 2005. The TREC interactive tracks: Putting the user into search. In Voorhees and Harman, editors, TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA.
• 4: W. Hersh and P. Over. 2001. Interactivity at the Text Retrieval Conference (TREC). IP&M, 37(3):365--367.
• 5: W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. 2005. TREC 2005 Genomics Track overview. In TREC 2005.
• 6: Jimmy Lin , Dina Demner-Fushman, Will pyramids built of nuggets topple over?, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.383-390, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220884]
• 7: A. Spink. 1997. Study of interactive feedback during mediated information retrieval. JASIS, 48(5):382--394.
• 8: E. Voorhees and H. Dang. 2005. Overview of the TREC 2005 question answering track. In TREC 2005.
• 9: E. Voorhees. 2003. Overview of the TREC 2003 question answering track. In TREC 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)