Exploring the limits of single-iteration clarification dialogs

ABSTRACT

Single-iteration clarification dialogs, as implemented in the TREC HARD track, represent an attempt to introduce interaction into ad hoc retrieval, while preserving the many benefits of large-scale evaluations. Although previous experiments have not conclusively demonstrated performance gains resulting from such interactions, it is unclear whether these findings speak to the nature of clarification dialogs, or simply the limitations of current systems. To probe the limits of such interactions, we employed a human intermediary to formulate clarification questions and exploit user responses. In addition to establishing a plausible upper bound on performance, we were also able to induce an "ontology of clarifications" to characterize human behavior. This ontology, in turn, serves as the input to a regression model that attempts to determine which types of clarification questions are most helpful. Our work can serve to inform the design of interactive systems that initiate user dialogs.

REFERENCES

• 1: E. Abels. The e-mail reference interview. RQ, 35(3):345--358, 1996.
• 2: J. Allan. HARD track overview in TREC 2003/2004/2005: High accuracy retrieval from documents. In TREC 2003/2004/2005.
• 3: James Allan , Ben Carterette , Joshua Lewis, When will information retrieval be "good enough"?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076109]
• 4: M. Bates. The Berry-Picking search: User interface design. In M. Dillon, editor, Interfaces for Information Retrieval and Online Systems: The State of the Art, pages 51--61. Greenwood Press, New Jersey, 1991.
• 5: C. Buckley and D. Harman. Reliable information access final workshop report, 2004.
• 6: Hang Cui , Renxu Sun , Keya Li , Min-Yen Kan , Tat-Seng Chua, Question answering passage retrieval using dependency relations, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076103]
• 7: B. Dervin and P. Dewdney. Neutral questioning: A new approach to the reference interview. RQ, 25(4):506--513, 1986.
• 8: Stephen P. Harter, Online information retrieval: concepts, principles, and techniques, Academic Press Professional, Inc., San Diego, CA, 1986
• 9: M. Hearst. Improving full-text precision on short queries using simple constraints. In Proceedings the Fifth Annual Symposium on Document Analysis and Information Retrieval (SDAIR 1996), 1996.
• 10: Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
• 11: William Hersh , Andrew Turpin , Susan Price , Benjamin Chan , Dale Kramer , Lynetta Sacherek , Daniel Olson, Do batch and user evaluations give the same results?, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.17-24, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345539]
• 12: Jürgen Koenemann , Nicholas J. Belkin, A case for interaction: a study of interactive information retrieval behavior and effectiveness, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.205-212, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238487]
• 13: Dawn Lawrie , W. Bruce Croft , Arnold Rosenberg, Finding topic words for hierarchical summarization, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.349-357, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384022]
• 14: Gary Marchionini, Information seeking in electronic environments, Cambridge University Press, New York, NY, 1995
• 15: Ragnar Nordlie, “User revealment”—a comparison of initial queries and ensuing question development in online searching and in human reference interactions, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.11-18, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312618]
• 16: T. Saracevic. Relevance: A review of and a framework for thinking on the notion in information science. JASIS, 26(6):321--343, 1975.
• 17: Amanda Spink , Abby Goodrum , David Robins , Mei Mei Wu, Elicitations during information retrieval: implications for IR system design, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243239]
• 18: Amanda Spink , Tefko Saracevic, Interaction in information retrieval: selection and effectiveness of search terms, Journal of the American Society for Information Science, v.48 n.8, p.741-761, Aug. 1997 [doi>10.1002/(SICI)1097-4571(199708)48:8<741::AID-ASI7>3.0.CO;2-S]
• 19: K. Swigger. Questions in library and information science. Lib. and Info. Sci. Research, 7:369--383, 1985.
• 20: R. Taylor. The process of asking questions. American Documentation, 13(4):391--396, 1962.
• 21: Ellen M. Voorhees, Query expansion using lexical-semantic relations, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.61-69, July 03-06, 1994, Dublin, Ireland
• 22: E. Voorhees and D. Harman. Overview of the Eighth Text REtrieval Conference (TREC-8). In TREC-8.
• 23: M. White. Questions in reference interviews. J. of Documentation, 54(4):443--465, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
TREC HARD, interactive retrieval, intermediated search