When will information retrieval be "good enough"?

ABSTRACT

We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators. Subjects performed the task by using these ranked lists to guide their labeling of answer passages in the retrieved articles. We found that as system accuracy improves, subject time on task and error rate decrease, and the rate of finding new correct answers increases. There is a large intermediary region in which the utility difference is not significant; our results suggest that there is some threshold of accuracy for this task beyond which user utility improves rapidly, but more experiments are needed to examine the area around that threshold closely.

REFERENCES

• 1: N. Abdul-Jaleel, J. Allan, W. Croft, F. Diaz, L. Larkey, X. Li, D. Metzler, M. Smucker, T. Strohman, H. Turtle, and C. Wade. UMass at TREC 2004: Notebook. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 657--670, 2004. Available online at http://trec.nist.gov.
• 2: J. Allan. HARD track overview in TREC 2004 (notebook), high accuracy retrieval from documents. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 226--235, 2004. Available online at http://trec.nist.gov.
• 3: Apache Software Foundation. Jakarta Lucene. Can be found at http://jakarta.apache.org/lucene/.
• 4: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 5: S. Boyce and A. Gorin. User interface issues for natural spoken dialog systems. In Proceedings of the International Symposium on Spoken Dialogue 96, pages 65--68, 1996.
• 6: Chris Buckley, Why current IR engines fail, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009132]
• 7: Chris Buckley , Ellen M. Voorhees, Retrieval evaluation with incomplete information, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009000]
• 8: James P. Callan, Passage-level evidence in document retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.302-310, July 03-06, 1994, Dublin, Ireland
• 9: Zhiwei Chen , Yunjie Xu, User-Oriented Relevance Judgment: A Conceptual Model, Proceedings of the Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 4, p.101.2, January 03-06, 2005 [doi>10.1109/HICSS.2005.654]
• 10: Nils Dahlbäck , Arne Jönsson , Lars Ahrenberg, Wizard of Oz studies: why and how, Proceedings of the 1st international conference on Intelligent user interfaces, p.193-200, January 04-07, 1993, Orlando, Florida, United States [doi>10.1145/169891.169968]
• 11: O. Drori and N. Alon. Using documents classification for displaying search results list. Technical Report 2002-46, Leibniz Center for Research in Computer Science, Hebrew University, Jerusalem, Israel, 2002.
• 12: Marcin Kaszkiel , Justin Zobel, Passage retrieval revisited, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.178-185, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 13: A. Kehler, J. Martin, A. Cheyer, L. Julia, J. Hobbs, and J. Bear. On representing salience and reference in multimodal human-computer interaction. In AAAI '98 (Representations for Multi-Model Human Computer Interaction), pages 33--39, 1998.
• 14: Jürgen Koenemann , Nicholas J. Belkin, A case for interaction: a study of interactive information retrieval behavior and effectiveness, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.205-212, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238487]
• 15: P. Over. TREC-6 interactive track report. In E. Voorhees and D. Harman, editors, The Sixth Text Retrieval Conference (TREC 6), pages 73--82, 1998. NIST Special Publication 500--240.
• 16: Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 17: I. Soboroff. Overview of the TREC 2004 novelty track. In E. Voorhees, editor, The Thirteenth Text Retrieval Conference (TREC 2004) Notebook, pages 57--70, 2004.
• 18: R. Swan and J. Allan. Improving interactive information retrieval effectiveness with 3-d graphics. Technical Report IR-100, University of Massachusetts, Amherst, Center for Intelligent Information Retrieval, 1996.
• 19: E. Tanin, A. Lotem, I. Haddadin, B. Shneiderman, C. Plaisant, and L. Slaughter. Facilitating network data exploration with query previews: A study of user performance and preference. Technical Report CS-TR-3879, University of Maryland, College Park, 1998.
• 20: E. Voorhees. Overview of the TREC 2003 question answering track. In E. Voorhees, editor, The Twelfth Text Retrieval Conference (TREC 2003), pages 73--82, 2003. NIST Special Publication 500--255.
• 21: William J. Weiland , Ben Schneiderman, A graphical query interface based on aggregation/generalization hierarchies, Information Systems, v.18 n.4, p.215-232, June 1993 [doi>10.1016/0306-4379(93)90018-V]
• 22: Mingfang Wu , Michael Fuller , Ross Wilkinson, Searcher performance in question answering, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.375-381, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384028]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing; Human factors

General Terms:
Design, Experimentation, Human Factors, Performance

Keywords:
information retrieval, passage retrieval, performance evaluation, user study