Bridging the lexical chasm: statistical approaches to answer-finding

ABSTRACT

This paper investigates whether a machine can automatically learn the task of finding, within a large collection of candidate responses, the answers to questions. The learning process consists of inspecting a collection of answered questions and characterizing the relation between question and answer with a statistical model. For the purpose of learning this relation, we propose two sources of data: Usenet FAQ documents and customer service call-center dialogues from a large retail company. We will show that the task of “answer-finding” differs from both document retrieval and tradition question-answering, presenting challenges different from those found in these problems. The central aim of this work is to discover, through theoretical and empirical investigation, those statistical techniques best suited to the answer-finding problem.

REFERENCES

• 1: AAAI. Proceedings of the AAAl FSS on Question Answering Systems (Cape Cod, MA, November 1999).
• 2: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 3: Peter F. Brown , John Cocke , Stephen A. Della Pietra , Vincent J. Della Pietra , Fredrick Jelinek , John D. Lafferty , Robert L. Mercer , Paul S. Roossin, A statistical approach to machine translation, Computational Linguistics, v.16 n.2, p.79-85, June 1990
• 4: Robin D. Burke , Kristian J. Hammond , Vladimir A. Kulyukin , Steven L. Lytinen , N. Tomuro , S. Schoenberg, Question Answering from Frequently Asked Question Files: Experiences with the FAQ Finder System, University of Chicago, Chicago, IL, 1997
• 5: Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T., and Harshman, R. Indexing by latent semantic analysis. Journal of the American Society for Information Science 41, 6 (1990), 391-407.
• 6: Dempster, A., Laird, N., and Rubin, D. Maximum likefihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society 39 (1977), 1-38.
• 7: Efthimiadis, E., and Biron, P. UCLA-Okapi at TREC- 2: Query expansion experiments. In Proceedings of the Second Text Retrieval Conference (1994).
• 8: GARTNER GROUP. Gartner group report, 1998.
• 9: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]
• 10: Lehnert, W. The process of question answering: A computer simulation of cognition. Lawrence Erlbaum Associates, 1978.
• 11: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 12: Weaver, W. Translation (1949). In Machine Translation of Languages. MIT Press, 1955.
• 13: Jinxi Xu , W. Bruce Croft, Query expansion using local and global document analysis, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.4-11, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243202]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  G. Mathematics of Computing

General Terms:
Algorithms, Experimentation, Human Factors, Theory