The search problem posed by large heterogeneous data sets in litigation: possible future approaches to research

ABSTRACT

Lawyers and their large institutional clients increasingly face the enormous problem of how to efficiently and efficaciously conduct searches for relevant documents in large heterogeneous electronic data sets, for the purpose of responding to litigation demands. Past research indicates that lawyers greatly overestimate their true rate of recall in civil discovery. The unprecedented size, scale, and complexity of electronically stored data now potentially subject to routine capture in litigation, for purpose of preservation, access, and review, presents information retrieval researchers with a series of important challenges to overcome. This paper describes the current context of e-discovery and discusses the potential for IR and AI research to address the challenges of conducting e-discovery. The TREC Legal Track is presented as a forum for the evaluation of e-discovery research and one new evaluation measure, elusion, is described, which has potential for addressing problems of measuring recall.

REFERENCES

• 1: ACE, http://www.nist.gov/speech/tests/ace/
• 2: Baron, J. R., with Oard, D.. M., Lewis, D. L., and Thompson, P. Supporting Search and Sensemaking in Electronically Stored Information for Discovery Proceedings, ICAIL 2007 Workshop Description, http://www.umiacs.umd.edu/~oard/desi-ws/.
• 3: Baron, J. R., Toward A Federal Benchmarking Standard for Evaluating Information Retrieval Products Used in E-Discovery, 6 Sedona Conference Journal (2005), 237--246 (available on Westlaw, Lexis)
• 4: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 5: David Blair, Wittgenstein, Language and Information: "Back to the Rough Ground!" (Information Science and Knowledge Management), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 6: Branting, K., The Role of Syntactic Analysis in Textual Case Retrieval, ICCBR Workshops 2005.
• 7: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148284]
• 8: CEAS. 2007, http://www.ceas.cc/
• 9: Collaborative Expedition Workshop #45, Advancing Information Sharing, Access, Discovery and Assimilation of Diverse Digital Collections Governed by Heterogeneous Sensitivities, held Nov. 8, 2005, see http://colab.cim3.net/cgi-bin/wiki.pl? AdvancingInformationSharing_DiverseDigitalCollections_HeterogeneousSensitivities_11_08_05
• 10: DARPA Proceedings Tipster Text Program Phase III October 1996 - October 1998.
• 11: GALE, 2007. http://www.darpa.mil/ipto/Programs/gale/index.htm
• 12: Ralph Grishman , Beth Sundheim, Message Understanding Conference-6: a brief history, Proceedings of the 16th conference on Computational linguistics, August 05-09, 1996, Copenhagen, Denmark [doi>10.3115/992628.992709]
• 13: ICAIL 2005. http://dale.liacs.nl/resources/misc/icail2005.pdf
• 14: Jensen, J. H., Special Issues Involving Electronic Discovery, 9 Kansas Journal of Law and Public Policy 425 (2000) (available on Westlaw, Lexis).
• 15: M. E. Maron , J. L. Kuhns, On Relevance, Probabilistic Indexing and Information Retrieval, Journal of the ACM (JACM), v.7 n.3, p.216-244, July 1960 [doi>10.1145/321033.321035]
• 16: Marie-Francine Moens, Combining structured and unstructured information in a retrieval model for accessing legislation, Proceedings of the 10th international conference on Artificial intelligence and law, June 06-11, 2005, Bologna, Italy [doi>10.1145/1165485.1165507]
• 17: Murphy, B., Special Issue: Readings on Case-based reasoning: Believe It - eDiscovery Technology Spending To Top $4.8 Billion By 2011, http://www.forrester.com/Research/Document/Excerpt/0,7211,40619,00.html.
• 18: Paul, G. L., and Baron, J. R. Information Inflation: Can The Legal System Adapt?, 13 Richmond Journal of Law and Technology 10 (2007), see law.richmond.edu/jolt/v13i3/article10.pdf
• 19: Roitblat, H., Appendix A to The Sedona Conference Draft Commentary on Search and Retrieval, Version 1.0 (April 2006) (unpublished draft available from the authors).
• 20: Gerard Salton, Manipulation of trees in information retrieval, Communications of the ACM, v.5 n.2, p.103-114, Feb. 1962 [doi>10.1145/366792.366828]
• 21: The Sedona Conference, The Sedona Principles: Best Practices Recommendations & Principles for Addressing Electronic Document Production (2005 version), see http://www.thesedonaconference.org/content/miscFiles/publications_html
• 22: SIAM, 2005. International Conference on Data Mining Workshop on Link Analysis, Counterterrorism and Security. http://www.cs.queensu.ca/~skill/siamworkshop.html
• 23: TREC 2006 Legal Track Overview, Baron, J. R., Oard, D. W., Lewis, D. D. available at http://trec-legal.umiacs.umd.edu/
• 24: Howard Turtle , W. Bruce Croft, Evaluation of an inference network-based retrieval model, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.187-222, July 1991 [doi>10.1145/125187.125188]
• 25: Voorhees, E., Overview of TREC 2005, The Fourteenth Annual Text Retrieval Conference (NIST 2005)
• 26: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005
• 27: Rosina O. Weber , Kevin D. Ashley , Stefanie Brüninghaus, Textual case-based reasoning, The Knowledge Engineering Review, v.20 n.3, p.255-260, September 2005 [doi>10.1017/S0269888906000713]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation

Keywords:
e-discovery, evaluation