Automatic categorization of case law

ABSTRACT

This paper describes a series of automatic text categorization experiments with case law documents. Cases are categorized into 40 broad, high-level categories. These results are compared to an existing operational process using Boolean queries manually constructed by domain experts. In this categorization process recall is considered more important than precision. This paper investigates three algorithms that potentially could automate this categorization process: 1) a nearest neighbor-like algorithm, 2) C4.5rules, a machine learning decision tree algorithm; and 3) Ripper, a machine learning rule induction algorithm. The results obtained by Ripper surpass those of the operational process.

REFERENCES

• 1: Chidanand Apté , Fred Damerau , Sholom M. Weiss, Towards language independent automated learning of text categorization models, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.23-30, July 03-06, 1994, Dublin, Ireland
• 2: Broglio, J.; Callan, J. P.; Croft, W. B.. INQUERY system overview In Proceedings of the TIPSTER TEXT PROGRAM (Phase 1) San Francisco: Morgan Kaufmann, 47-67, 1994.
• 3: Stefanie Brüninghaus , Kevin D. Ashley, Toward adding knowledge to learning algorithms for indexing legal cases, Proceedings of the 7th international conference on Artificial intelligence and law, p.9-17, June 14-17, 1999, Oslo, Norway [doi>10.1145/323706.323709]
• 4: Stefanie Brüninghaus , Kevin D. Ashley, Finding factors: learning to classify case opinions under abstract fact categories, Proceedings of the 6th international conference on Artificial intelligence and law, p.123-131, June 30-July 03, 1997, Melbourne, Australia [doi>10.1145/261618.261645]
• 5: Cohen, W. W. Efficient pruning methods for separate-and-conquer rule learning systems. In Proceedings of the 13 th International Joint Conference on Artificial Intelligence (Chambery, France), 1993.
• 6: Cohen, W. W. Fast effective rule induction Machine Learning: Proceedings of the Twelfth International Conference San Francisco: Morgan Kaufmann, 1995.
• 7: William W. Cohen , Yoram Singer, Context-sensitive learning methods for text categorization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.307-315, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243278]
• 8: William W. Cohen , Yoram Singer, Context-sensitive learning methods for text categorization, ACM Transactions on Information Systems (TOIS), v.17 n.2, p.141-173, April 1999 [doi>10.1145/306686.306688]
• 9: Cooper, W. S.. Expected search length: a single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation 19, 1, 30-41, 1968.
• 10: Crawford, S. L.; Fung, R.; Appelbaum, L. A.; Tong, R. M. Classification trees for information retrieval In Birnbaum, L. A. and Collins, G. C. (eds.) Machine Learning Proceedings of the Eighth International Workshop (ML91) San Mateo, CA: Morgan Kaufmann, 245-249.
• 11: Robert H. Creecy , Brij M. Masand , Stephen J. Smith , David L. Waltz, Trading MIPS and memory for knowledge engineering, Communications of the ACM, v.35 n.8, p.48-64, Aug. 1992 [doi>10.1145/135226.135228]
• 12: Curran, T. and Thompson, P. Automatic Categorization of Statute Documents Proceedings of the 8th ASIS SIG/CR Classification Research Workshop (Washington, D.C1997), 19-30.
• 13: Jody J. Daniels , Edwina L. Rissland, Finding legally relevant passages in case opinions, Proceedings of the 6th international conference on Artificial intelligence and law, p.39-46, June 30-July 03, 1997, Melbourne, Australia [doi>10.1145/261618.261627]
• 14: Danet, B. Language in the legal process. Law & Society Review 14, 3, 445-564, 1980.
• 15: Philip J. Hayes, Intelligent high-volume text processing using shallow, domain-specific techniques, Text-based intelligent systems: current research and practice in information extraction and retrieval, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1992
• 16: Julia Hodges , Shiyun Yie , Ray Reighart , Lois Boggess, An automated system that assists in the generation of document indexes, Natural Language Engineering, v.2 n.2, p.137-160, June 1996 [doi>10.1017/S1351324996001325]
• 17: Kittredge, R. and Lehrberger, J. (eds.). Sublanguage: studies of language in a restricted domain. de Gruyter, New York, 1982.
• 18: Daphne Koller , Mehran Sahami, Hierarchically Classifying Documents Using Very Few Words, Proceedings of the Fourteenth International Conference on Machine Learning, p.170-178, July 08-12, 1997
• 19: Chi-Hong Leung , Wing-Kay Kan, A statistical learning approach to automatic indexing of controlled index terms, Journal of the American Society for Information Science, v.48 n.1, p.55-66, Jan. 1997 [doi>10.1002/(SICI)1097-4571(199701)48:1<55::AID-ASI7>3.3.CO;2-9]
• 20: David Dolan Lewis, Representation and learning in information retrieval, University of Massachusetts, Amherst, MA, 1992
• 21: David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
• 22: Lewis, D. D. and Ringuette, M. A comparison of two learning algorithms for text categorization. Proceedings of the Third Annual Symposium on Document Analysis and Information Retrieval Las Vegas, Nevada: Information Science Research Institute, 81-93, 1994.
• 23: David D. Lewis , Robert E. Schapire , James P. Callan , Ron Papka, Training algorithms for linear text classifiers, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.298-306, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243277]
• 24: M. E. Maron, Automatic Indexing: An Experimental Inquiry, Journal of the ACM (JACM), v.8 n.3, p.404-417, July 1961 [doi>10.1145/321075.321084]
• 25: Brij Masand , Gordon Linoff , David Waltz, Classifying news stories using memory based reasoning, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.59-65, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133177]
• 26: Moulinier, I. Feature selection: a Useful preprocessing step. 19th Annual BCS-IRSG Colloquium on IR Research, 1-11, 1997.
• 27: Moulinier, I., Raskinis, G., Ganascia, J.-G. Text categorization: a symbolic approach. In Information Science Research Institute, University of Nevada, Las Vegas (Ed.). Proceedings Fifth Annual Symposium on Document Analysis and Information Retrieval (). Las Vegas: University of Nevada, Las Vegas, 87-99, 1996.
• 28: Anandeep S. Pannu, Using genetic algorithms to inductively reason with cases in the legal domain, Proceedings of the 5th international conference on Artificial intelligence and law, p.175-184, May 21-24, 1995, College Park, Maryland, United States [doi>10.1145/222092.222232]
• 29: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 30: Quinlan, J. R. Improved Use of Continuous Attributes in C4.5 Journal of Artificial Intelligence Research. 4, 77-90, 1996.
• 31: Quinlan, R. http://www.rulequest.com, 1997.
• 32: Rissland, E. L. and Daniels, J. J. Using CBR to drive IR In International Joint Conference on Artificial Intelligence (IJCAI-95) (Montreal, Canada), 400-407.
• 33: Schweighofer, E. and Winiwarter, W. Intelligent information retrieval: KONTERM - Automatic representation of context related terms within a knowledge base for a legal expert system. Proceedings of the 25th Anniversary Conference of the Istituto per la documentazione giuridica of the CNR: Towards a Global Expert System in Law, (Padua, Italy), 1994
• 34: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 35: West Publishing Company. WESTLAW Reference Manual, 5th ed. West Publishing Company, St. Paul MN, 1993.
• 36: West Publishing Company. West's Analysis of American Law. West Publishing Company, St. Paul MN, 1994.
• 37: West Publishing Company. West's Law Finder: A Legal Resources Guide. West Publishing Company, St. Paul, MN, 1995.
• 38: Yang, Y. Sampling strategies and learning efficiencies in text categorization. In AAAI Spring Symposium on Machine Learning in Information Access, 88-95, 1996.
• 39: Yang, Y. An evaluation of statistical approaches to text categorization. Carnegie Mellon University School of Computer Science technical report CMU- CS-97-127, 1997.
• 40: Yiming Yang , Christopher G. Chute, An application of least squares fit mapping to text information retrieval, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.281-290, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160738]
• 41: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.2 Information Storage
          Subjects: Record classification**

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY

General Terms:
Design, Experimentation, Theory

Keywords:
text categorization