A study in rule-specific issue categorization for e-rulemaking

ABSTRACT

We address the e-rulemaking problem of categorizing public comments according to the issues that they address. In contrast to previous text categorization research in e-rulemaking [5, 6], and in an attempt to more closely duplicate the comment analysis process in federal agencies, we employ a set of rule-specific categories, each of which corresponds to a significant issue raised in the comments. We describe the creation of a corpus to support this text categorization task and report interannotator agreement results for a group of six annotators. We outline those features of the task and of the e-rulemaking context that engender both a non-traditional text categorization corpus and a correspondingly difficult machine learning problem. Finally, we investigate the application of standard and hierarchical text categorization techniques to the e-rulemaking data sets and find that automatic categorization methods show promise as a means of reducing the manual labor required to analyze large comment sets: the automatic annotation methods approach the performance of human annotators for both flat and hierarchical issue categorization.

REFERENCES

• 1: Susan Dumais , Hao Chen, Hierarchical classification of Web content, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.256-263, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345593]
• 2: V. Hatzivassiloglou, J. Klavans, and E. Eskin. Detecting Text Similarity over Short Passages: Exploring Linguistic Feature Combinations via Machine Learning. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-99), pages 203--212, University of Maryland, College Park, MD, 1999. Association for Computational Linguistics.
• 3: Thorsten Joachims, Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms, Kluwer Academic Publishers, Norwell, MA, 2002
• 4: K. Krippendorff. Content analysis: An introduction to its methodology (2nd Ed.). Sage Publications, 2004.
• 5: Namhee Kwon , Eduard Hovy, Information acquisition using multiple classifications, Proceedings of the 4th international conference on Knowledge capture, October 28-31, 2007, Whistler, BC, Canada [doi>10.1145/1298406.1298427]
• 6: Namhee Kwon , Stuart W. Shulman , Eduard Hovy, Multidimensional text analysis for eRulemaking, Proceedings of the 2006 international conference on Digital government research, May 21-24, 2006, San Diego, California [doi>10.1145/1146598.1146649]
• 7: Mehran Sahami , Timothy D. Heilman, A web-based kernel function for measuring the similarity of short text snippets, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135834]
• 8: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 9: D. Silver, G. Bakir, K. Bennett, R. Caruana, M. Pontil, S. Russell, and P. Tadepalli. Inductive Transfer : 10 Years Later. NIPS 2005 Workshop, 2005.
• 10: P. Strauss, T. Rakoff, and C. Farina. Administrative Law. 10th edition, 2003.
• 11: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 12: Hui Yang , Jamie Callan, Near-duplicate detection for eRulemaking, Proceedings of the 2005 national conference on Digital government research, May 15-18, 2005, Atlanta, Georgia
• 13: Hui Yang , Jamie Callan, Near-duplicate detection by instance-level constrained clustering, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148243]
• 14: Sarah Zelikovitz , Haym Hirsh, Improving Short-Text Classification using Unlabeled Data for Classification Problems, Proceedings of the Seventeenth International Conference on Machine Learning, p.1191-1198, June 29-July 02, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
      I.2.7 Natural Language Processing
          Subjects: Text analysis
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

  J. Computer Applications
  J.1 ADMINISTRATIVE DATA PROCESSING
      Subjects: Government

General Terms:
Algorithms, Design, Management