Toward automatic facet analysis and need negotiation

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Toward automatic facet analysis and need negotiation: Lessons from mediated search

Full text

Pdf (610 KB)

Source

ACM Transactions on Information Systems (TOIS) archive
Volume 27 , Issue 1 (December 2008) table of contents

Article No. 6

Year of Publication: 2008

ISSN:1046-8188

Authors

Jimmy Lin	University of Maryland, College Park, College Park, MD
Philip Wu	University of Maryland, College Park, College Park, MD
Eileen Abels	Drexel University, Philadelphia, PA

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 18, Downloads (12 Months): 269, Citation Count: 0

Additional Information:

abstract references index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1416950.1416956 What is a DOI?

ABSTRACT

This work explores the hypothesis that interactions between a trained human search intermediary and an information seeker can inform the design of interactive IR systems. We discuss results from a controlled Wizard-of-Oz case study, set in the context of the TREC 2005 HARD track evaluation, in which a trained intermediary executed an integrated search and interaction strategy based on conceptual facet analysis and informed by need negotiation techniques common in reference interviews. Having a human “in the loop” yielded large improvements over fully automated systems as measured by standard ranked-retrieval metrics, demonstrating the value of mediated search. We present a detailed analysis of the intermediary's actions to gain a deeper understanding of what worked and why. One contribution is a taxonomy of clarification types informed both by empirical results and existing theories in library and information science. We discuss how these findings can guide the development of future systems. Overall, this work illustrates how studying human information-seeking processes can lead to better information retrieval applications.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Abels, E. G. 1996. The e-mail reference interview. Ref. Quart. 35, 3, 345--358.
	2	Allan, J. 2003. High accuracy retrieval from documents (Hard) track overview. In Proceedings of the Twelfth Text REtrieval Conference (TREC). 24--37.
	3	Allan, J. 2005. High accuracy retrieval from documents (HARD) track overview. In Proceedings of the Fourteenth Text REtrieval Conference (TREC). 51--67.
	4	James Allan , Ben Carterette , Joshua Lewis, When will information retrieval be "good enough"?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076109]
	5	Bates, M. J. 1991. The Berry-Picking search: User interface design. In Interfaces for Information Retrieval and Online Systems: The State of the Art, M. Dillon, Ed. Greenwood Press, Westport, CT, 51--61.
	6	Beaulieu, M., Do, T., Payne, A., and Jones, S. 1997. ENQUIRE Okapi Project. British Library Research and Innovation Report 17.
	7	Micheline Beaulieu , Stephen Robertson , Edie Rasmussen, Evaluating interactive systems in TREC, Journal of the American Society for Information Science, v.47 n.1, p.85-94, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<85::AID-ASI8>3.3.CO;2-7]
	8	Belkin, N. J., Cool, C., Head, J., Jeng, J., Kelly, D., jeng Lin, S., Lobash, L., Park, S., Savage-Knepshield, P. A., and Sikora, C. 1999. Relevance feedback versus Local Context Analysis as term suggestion devices: Rutgers' TREC-8 interactive track experience. In Proceedings of the Eighth Text REtrieval Conference, (TREC-8). 565--573.
	9	Belkin, N. J., Cool, C., Stein, A., and Thiel, U. 1995. Cases, scripts and information-seeking strategies: On the design of interactive information retrieval systems. Expert Syst. Appl. 9, 3, 379--395.
	10	Matthew W. Bilotti , Paul Ogilvie , Jamie Callan , Eric Nyberg, Structured retrieval for question answering, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277802]
	11	David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
	12	Bopp, R. E. and Smith, L. C. 1995. Reference and Information Services: An Introduction, 2nd ed. Libraries Unlimited, Englewood, CO.
	13	Peter Bruza , Robert McArthur , Simon Dennis, Interactive Internet search: keyword, directory and query reformulation mechanisms compared, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.280-287, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345598]
	14	Jean Carletta, Assessing agreement on classification tasks: the kappa statistic, Computational Linguistics, v.22 n.2, June 1996
	15	Claudio Carpineto , Renato de Mori , Giovanni Romano , Brigitte Bigi, An information-theoretic approach to automatic query expansion, ACM Transactions on Information Systems (TOIS), v.19 n.1, p.1-27, Jan. 2001 [doi>10.1145/366836.366860]
	16	Cleverdon, C. W., Mills, J., and Keen, E. M. 1968. Factors determining the performance of indexing systems. Two volumes. ASLIB Cranfield Research Project, Cranfield, England.
	17	B. Croft, An approach to natural language for document retrieval, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.26-32, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42009]
	18	Hang Cui , Renxu Sun , Keya Li , Min-Yen Kan , Tat-Seng Chua, Question answering passage retrieval using dependency relations, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076103]
	19	Dina Demner-Fushman , Jimmy Lin, Answer extraction, semantic clustering, and extractive summarization for clinical question answering, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, p.841-848, July 17-18, 2006, Sydney, Australia [doi>10.3115/1220175.1220281]
	20	Dina Demner-Fushman , Jimmy Lin, Answering Clinical Questions with Knowledge-Based and Statistical Techniques, Computational Linguistics, v.33 n.1, p.63-103, March 2007 [doi>10.1162/coli.2007.33.1.63]
	21	Dervin, B. 1991. From the mind's eye of the user: The sense-making qualitative-quantitative methodology. In Qualitative Research in Information Management, J. D. Glazier and R. R. Powell, Eds. Libraries Unlimited, Inc., Englewood, CO, 61--84.
	22	Dervin, B. and Dewdney, P. 1986. Neutral questioning: A new approach to the reference interview. Ref. Quart. 25, 4, 506--513.
	23	Susan Dumais , Edward Cutrell , Hao Chen, Optimizing search by showing results in context, Proceedings of the SIGCHI conference on Human factors in computing systems, p.277-284, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365116]
	24	Efthimis N. Efthimiadis, Interactive query expansion: a user-based evaluation in a relevance feedback environment, Journal of the American Society for Information Science, v.51 n.11, p.989-1003, Sept. 2000 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1002>3.0.CO;2-B]
	25	Efthimiadis, E. N. and Robertson, S. E. 1989. Feedback and interaction in information retrieval. In Perspectives in Information Management, C. Oppenheim, Ed. Butterworth, London, England, 257--272.
	26	Valerie Florance , Gary Marchionini, Information processing in the context of medical care, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.158-163, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215353]
	27	Francoeur, S. 2001. An analytical survey of chat reference services. Ref. Serv. Rev. 29, 3, 189--204.
	28	Jianfeng Gao , Jian-Yun Nie , Guangyuan Wu , Guihong Cao, Dependence language model for information retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009024]
	29	Sanda Harabagiu , Andrew Hickl , John Lehmann , Dan Moldovan, Experiments with interactive question-answering, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.205-214, June 25-30, 2005, Ann Arbor, Michigan [doi>10.3115/1219840.1219866]
	30	Sanda M. Harabagiu , Marius A. Paşca , Steven J. Maiorano, Experiments with open-domain textual Question Answering, Proceedings of the 18th conference on Computational linguistics, p.292-298, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990863]
	31	D. Harman, Towards interactive query expansion, Proceedings of the 11th annual international ACM SIGIR conference on Research and development in information retrieval, p.321-331, May 1988, Grenoble, France [doi>10.1145/62437.62469]
	32	Harman, D. K. 2005. The TREC test collections. In TREC: Experiment and Evaluation in Information Retrieval, E. M. Voorhees and D. K. Harman, Eds. MIT Press, Cambridge, MA, 21--52.
	33	Stephen P. Harter, Online information retrieval: concepts, principles, and techniques, Academic Press Professional, Inc., San Diego, CA, 1986
	34	Hawkins, D. T. and Wagers, R. 1982. Online bibliographic search strategy development. Online 6, 3, 12--19.
	35	Marti A. Hearst, TileBars: visualization of term distribution information in full text information access, Proceedings of the SIGCHI conference on Human factors in computing systems, p.59-66, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223912]
	36	Marti A. Hearst , Jan O. Pedersen, Reexamining the cluster hypothesis: scatter/gather on retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.76-84, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243216]
	37	Hersh, W. R. and Over, P. 2001. Interactivity at the Text Retrieval Conference (TREC). Inform. Process. Manage. 37, 3, 365--367.
	38	William Hersh , Andrew Turpin , Susan Price , Benjamin Chan , Dale Kramer , Lynetta Sacherek , Daniel Olson, Do batch and user evaluations give the same results?, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.17-24, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345539]
	39	Ingwersen, P. 1996. Cognitive perspectives of information retrieval interaction: Elements of a cognitive IR theory. J. Documentat. 52, 1, 3--50.
	40	Ingwersen, P. 1999. Cognitive information retrieval. Ann. Rev. Inform. Sci. Tech. 34, 1, 3--52.
	41	J. F. Kelley, An iterative design methodology for user-friendly natural language office information applications, ACM Transactions on Information Systems (TOIS), v.2 n.1, p.26-41, Jan. 1984 [doi>10.1145/357417.357420]
	42	Diane Kelly , Vijay Deepak Dollu , Xin Fu, The loquacious user: a document-independent source of terms for query expansion, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076112]
	43	Diane Kelly , Xin Fu, Elicitation of term relevance feedback: an investigation of term source and context, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148249]
	44	Knapp, S. D. 1978. The reference interview in the computer-based setting. Ref. Quart. 17, 4, 320--324.
	45	Jürgen Koenemann , Nicholas J. Belkin, A case for interaction: a study of interactive information retrieval behavior and effectiveness, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.205-212, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238487]
	46	Bill Kules , Jack Kustanowitz , Ben Shneiderman, Categorizing web search results into meaningful and stable categories using fast-feature techniques, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141801]
	47	Leuski, A. and Allan, J. 2000. Strategy-based interactive cluster visualization for information retrieval. Internat. J. Dig. Libr. 3, 2, 170--184.
	48	Jimmy Lin , Michael DiCuccio , Vahan Grigoryan , W. John Wilbur, Navigating information spaces: A case study of related article search in PubMed, Information Processing and Management: an International Journal, v.44 n.5, p.1771-1783, September, 2008 [doi>10.1016/j.ipm.2008.04.002]
	49	Jimmy Lin , Mark D. Smucker, How do users find things with PubMed?: towards automatic utility evaluation with user simulations, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390340]
	50	Lin, J. and Wilbur, W. J. 2007. PubMed related articles: A probabilistic topic-based model for content similarity. BMC Bioinform. 8, 423.
	51	Jimmy Lin , Philip Wu , Dina Demner-Fushman , Eileen Abels, Exploring the limits of single-iteration clarification dialogs, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148251]
	52	Gary Marchionini, Information seeking in electronic environments, Cambridge University Press, New York, NY, 1995
	53	Merriam, S. B. 1998. Qualitative Research and Case Study Applications in Education. Jossey-Bass, San Francisco, CA.
	54	Donald Metzler , W. Bruce Croft, A Markov random field model for term dependencies, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076115]
	55	Ragnar Nordlie, “User revealment”—a comparison of initial queries and ensuing question development in online searching and in human reference interactions, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.11-18, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312618]
	56	Paul Over, The TREC interactive track: an annotated bibliography, Information Processing and Management: an International Journal, v.37 n.3, p.369-381, May 2001 [doi>10.1016/S0306-4573(00)00053-4]
	57	Pirolli, P. and Card, S. 1999. Information foraging. Psychol. Rev. 106, 4, 643--675.
	58	John Prager, Open-domain question: answering, Foundations and Trends in Information Retrieval, v.1 n.2, p.91-231, January 2006 [doi>10.1561/1500000001]
	59	Ian Ruthven, Re-examining the potential effectiveness of interactive query expansion, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860475]
	60	Salton, G. and Buckley, C. 1990. Improving retrieval performance by relevance feedback. J. Amer. Soc. Inform. Sci. 41, 4, 288--297.
	61	Saracevic, T. 1975. Relevance: A review of and a framework for thinking on the notion in information science. J. Amer. Soc. Inform. Sci. 26, 6, 321--343.
	62	Saracevic, T. 1997a. The stratified model of information retrieval interaction: Extension and applications. In Proceedings of the 60th Annual Meeting of the American Society for Information Science and Technology.
	63	Tefko Saracevic, Users lost: reflections on the past, future, and limits of information science, ACM SIGIR Forum, v.31 n.2, p.16-27, Fall 1997 [doi>10.1145/270886.270889]
	64	Sharon Small , Tomek Strzalkowski , Ting Liu , Sean Ryan , Robert Salkin , Nobuyuki Shimizu , Paul Kantor , Diane Kelly , Robert Rittman , Nina Wacholder, HITIQA: towards analytical question answering, Proceedings of the 20th international conference on Computational Linguistics, p.1291-es, August 23-27, 2004, Geneva, Switzerland [doi>10.3115/1220355.1220544]
	65	P. J. Smith , S. J. Shute , D. Galdes, In search of knowledge-based search tactics, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-10, June 25-28, 1989, Cambridge, Massachusetts, United States
	66	Mark D. Smucker , James Allan, Find-similar: similarity browsing as a search tool, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148250]
	67	Eero Sormunen, Liberal relevance criteria of TREC -: counting on negligible documents?, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564433]
	68	Karen Sparck Jones, Further reflections on TREC, Information Processing and Management: an International Journal, v.36 n.1, p.37-85, Jan. 2000 [doi>10.1016/S0306-4573(99)00044-8]
	69	Spink, A. 1997. Study of interactive feedback during mediated information retrieval. J. Amer. Soc. Inform. Sci. 48, 5, 382--394.
	70	Amanda Spink , Abby Goodrum , David Robins , Mei Mei Wu, Elicitations during information retrieval: implications for IR system design, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243239]
	71	Amanda Spink , Howard Greisdorf, Regions and levels: measuring and mapping users' relevance judgments, Journal of the American Society for Information Science and Technology, v.52 n.2, p.161-173, Jan 2001 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1564>3.3.CO;2-C]
	72	Amanda Spink , Tefko Saracevic, Interaction in information retrieval: selection and effectiveness of search terms, Journal of the American Society for Information Science, v.48 n.8, p.741-761, Aug. 1997 [doi>10.1002/(SICI)1097-4571(199708)48:8<741::AID-ASI7>3.0.CO;2-S]
	73	Swanson, D. R. 1977. Information retrieval as a trial-and-error process. Libr. Quart. 47, 2, 128--148.
	74	Swigger, K. 1985. Questions in library and information science. Libr. Inform. Sci. Res. 7, 369--383.
	75	Taylor, R. S. 1962. The process of asking questions. Amer. Documentat. 13, 4, 391--396.
	76	Taylor, R. S. 1968. Question-negotiation and information seeking in libraries. Coll. Res. Libr. 29, 178--194.
	77	Toda, H., Kataoka, R., and Oku, M. 2007. The usefulness of dynamically categorizing search results. Int. J. Hum. Comput. Interact. 23, 1--2, 3--23.
	78	Andrew H. Turpin , William Hersh, Why batch and user evaluations do not give the same results, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.225-231, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383992]
	79	Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148176]
	80	Howard Turtle , W. Bruce Croft, Evaluation of an inference network-based retrieval model, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.187-222, July 1991 [doi>10.1145/125187.125188]
	81	Aravindan Veerasamy , Nicholas J. Belkin, Evaluation of a tool for visualization of information retrieval results, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.85-92, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243218]
	82	Voorhees, E. M. 2005. Overview of the TREC 2005 robust retrieval track. In Proceedings of the Fourteenth Text REtrieval Conference (TREC). 81--89.
	83	Voorhees, E. M. and Harman, D. K. 1999. Overview of the Eighth Text REtrieval Conference (TREC-8). In Proceedings of the Eighth Text REtrieval Conference. 1--23.
	84	Nina Wacholder , Diane Kelly , Paul Kantor , Robert Rittman , Ying Sun , Bing Bai , Sharon Small , Boris Yamrom , Tomek Strzalkowski, A model for quantitative evaluation of an end-to-end question-answering system, Journal of the American Society for Information Science and Technology, v.58 n.8, p.1082-1099, June 2007 [doi>10.1002/asi.v58:8]
	85	White, M. D. 1985. Evaluation of the reference interview. Ref. Quart. 25, 1, 76--83.
	86	White, M. D. 1998. Questions in reference interviews. J. Documentat. 54, 4, 443--465.
	87	W. John Wilbur , Leona Coffee, The effectiveness of document neighboring in search enhancement, Information Processing and Management: an International Journal, v.30 n.2, p.253-266, March-April 1994 [doi>10.1016/0306-4573(94)90068-X]
	88	Wilson, T. D. 1999. Models in information behaviour research. J. Documentat. 55, 3, 249--270.
	89	Ka-Ping Yee , Kirsten Swearingen , Kevin Li , Marti Hearst, Faceted metadata for image search and browsing, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642681]
	90	Yin, R. K. 2003. Case Study Research: Design and Methods. Sage Publications, Newbury Park, CA.
	91	Chengxiang Zhai , John Lafferty, Model-based feedback in the language modeling approach to information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502654]