Cost and benefit analysis of mediated enterprise search

ABSTRACT

The utility of an enterprise search system is determined by three key players: the information retrieval (IR) system (the search engine), the enterprise users, and the service provider who delivers the tailored IR service to its designated enterprise users. Currently, evaluations of enterprise search have been focused largely on the IR system effectiveness and efficiency, only a relatively small amount of effort on the user's involvement, and hardly any effort on the service provider's role. This paper will investigate the role of the service provider. We propose a method that evaluates the cost and benefit for a service provider of using a mediated search engine - in particular, where domain experts intervene on the ranking of the search results from a search engine. We test our cost and benefit evaluation method in a case study and conduct user experiments to demonstrate it. Our study shows that: 1) by making use of domain experts' relevance assessments in search result ranking, the precision and the discount cumulated gain of ranked lists have been improved significantly (144% and 40% respectively); 2) the service provider gains substantial return on investment and higher search success rate by investing in domain experts' relevance assessments; and 3) the cost and benefit evaluation also indicates the type of queries to be selected from a query log for evaluating an enterprise search engine.

REFERENCES

• 1: Rodrigo B. Almeida , Virgilio A. F. Almeida, A community-aware search engine, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988728]
• 2: Buckley, C., Salton, G. and Allan, J. (1992), Automatic retrieval with locality information using SMART, In Proceedings of the first Text Retrieval Conference, Gaithersburg, USA, pp.59--72.
• 3: DeLone, W. H. & McLean E. R. (1992), Information systems success: The quest for the dependent variable. Information Systems Research. v3(1), pp.60--96.
• 4: Fleiss, J. L. (1971), Measuring nominal scale agreement among many raters. Psychological Bulletin, v76(5), pp.378--382.
• 5: Zhiwei Guan , Edward Cutrell, An eye tracking study of the effect of target rank on web search, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240691]
• 6: Ingwersen, P. and Järvelin, K. (2004). Information retrieval in contexts. In Proceedings of the SIGIR 2004 IRiX workshop, Sheffield UK, pp.6--9.
• 7: Peter Ingwersen , Kalervo Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 8: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 9: Thorsten Joachims, Optimizing search engines using clickthrough data, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775067]
• 10: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007 [doi>10.1145/1229179.1229181]
• 11: Lesk, M. and Salton, G. (1968). Relevance assessments and retrieval system evaluation. Information Storage Retrieval. v4. pp.179--189.
• 12: Gheorghe Muresan , David J. Harper, Document Clustering and Language Models for System-Mediated Information Access, Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries, p.438-449, September 04-09, 2001
• 13: Ragnar Nordlie, “User revealment”—a comparison of initial queries and ensuing question development in online searching and in human reference interactions, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.11-18, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312618]
• 14: Nordlie, R. (1996). Unmediated and mediated information searching in the public library. In Proceedings of ASIS Annual Meeting, v33, pp.41--46.
• 15: O'Conner, J. (1969). Some independent agreements and resolved disagreements about answer-providing documents. American Documentation, v20, pp.311--319.
• 16: Paris, C., Colineau, N. and Wilkinson, R. (2007), NLG systems evaluation: a framework to measure impact on and cost for all stakeholders. Position Paper presented in Workshop on Shared Tasks and Comparative Evaluation in Natural Language Processing, Arlinton, Virginia, USA.
• 17: Paris, C., Colineau, N. and Wilkinson, R. (2007). Bang for buck in exploratory search. CSIRO ICT Centre Technical Report. Report Number: 09/197.
• 18: Robertson, S., Walker, S. Jones, S. Hancock-Beaulieu, M. and Gatford, M. (1994). Okapi at TREC-3. In Proceedings of the Third Text Retrieval Conference, Gaitherburg, USA.
• 19: Ian Ruthven , Mounia Lalmas, A survey on the use of relevance feedback for information access systems, The Knowledge Engineering Review, v.18 n.2, p.95-145, June 2003 [doi>10.1017/S0269888903000638]
• 20: Peter G. Sassone, Cost benefit analysis of information systems: a survey of methodologies, Proceedings of the ACM SIGOIS and IEEECS TC-OA 1988 conference on Office information systems, p.126-133, March 23-25, 1988, Palo Alto, California, United States
• 21: Peter G. Sassone, Cost-benefit methodology for office systems, ACM Transactions on Information Systems (TOIS), v.5 n.3, p.273-289, July 1987 [doi>10.1145/27641.28059]
• 22: Xuehua Shen , Bin Tan , ChengXiang Zhai, Context-sensitive information retrieval using implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076045]
• 23: Barry Smyth , Evelyn Balfe , Jill Freyne , Peter Briggs , Maurice Coyle , Oisin Boydell, Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine, User Modeling and User-Adapted Interaction, v.14 n.5, p.383-423, January 2005 [doi>10.1007/s11257-004-5270-4]
• 24: Amanda Spink , T. D. Wilson , Nigel Ford , Allen Foster , David Ellis, Information-seeking and mediated searching. Part 1: theoretical framework and research design, Journal of the American Society for Information Science and Technology, v.53 n.9, p.695-703, August 2002 [doi>10.1002/asi.10081]
• 25: Kazunari Sugiyama , Kenji Hatano , Masatoshi Yoshikawa, Adaptive web search based on user profile constructed without any effort from users, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988764]
• 26: Teevan, J., Dumais, S. and Horvitz, E. (2005). Beyond the commons: Investigating the value of personalizing web search. In Proceedings of PLA 2005: Workshops on New Technologies for Personalized Information Access. pp.82--92
• 27: White, R. W., Jose, J. M. and Ruthven, I. (2001), Comparing explicit and implicit feedback techniques for web retrieval: TREC-10 interactive track report. In Proceedings of the 10th Text Retrieval Conference, Gaithersburg, USA.
• 28: Ryen W. White , Diane Kelly, A study on the effects of personalization and task information on implicit feedback performance, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183659]
• 29: Mingfang Wu , Andrew Turpin , Justin Zobel, An investigation on a community's web search variability, Proceedings of the thirty-first Australasian conference on Computer science, January 01-01, 2008, Wollongong, Australia
• 30: Xie, Y. and O'Hallaron, D. (2002). Locality in Search Engine Queries and Its Implications for Caching. In Proceedings of IEEE INFOCOM 2002. pp.1238--1247

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.1 Systems and Information Theory
          Subjects: Value of information

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Economics, Human Factors, Measurement, Performance

Keywords:
cost and benefit analysis, enterprise search, evaluation, information retrieval, mediated search, relevance feedback