Active learning for e-rulemaking: public comment categorization

ABSTRACT

We address the e-rulemaking problem of reducing the manual labor required to analyze public comment sets. In current and previous work, for example, text categorization techniques have been used to speed up the comment analysis phase of e-rulemaking --- by classifying sentences automatically, according to the rule-specific issues [2] or general topics that they address [7, 8]. Manually annotated data, however, is still required to train the supervised inductive learning algorithms that perform the categorization. This paper, therefore, investigates the application of active learning methods for public comment categorization: we develop two new, general-purpose, active learning techniques to selectively sample from the available training data for human labeling when building the sentence-level classifiers employed in public comment categorization. Using an e-rulemaking corpus developed for our purposes [2], we compare our methods to the well-known query by committee (QBC) active learning algorithm [5] and to a baseline that randomly selects instances for labeling in each round of active learning. We show that our methods statistically significantly exceed the performance of the random selection active learner and the query by committee (QBC) variation, requiring many fewer training examples to reach the same levels of accuracy on a held-out test set. This provides promising evidence that automated text categorization methods might be used effectively to support public comment analysis.

REFERENCES

• 1: K. Brinker. Incorporating diversity in active learning with support vector machines. In Proceedings of ICML-03, 20th International Conference on Machine Learning. Morgan Kaufmann Publishers, San Francisco, US, 2003.
• 2: Claire Cardie , Cynthia Farina , Adil Aijaz , Matt Rawding , Stephen Purpura, A study in rule-specific issue categorization for e-rulemaking, Proceedings of the 2008 international conference on Digital government research, May 18-21, 2008, Montreal, Canada
• 3: C. Coglianese. Weak democracy, strong information: The role of information technology in the rulemaking process. In V. Mayer-Schoenberger and D. Lazer, editors, Electronic Government to Information Government: Governing in the 21ST Century, 2007.
• 4: David Cohn , Les Atlas , Richard Ladner, Improving Generalization with Active Learning, Machine Learning, v.15 n.2, p.201-221, May 1994 [doi>10.1023/A:1022673506211]
• 5: Yoav Freund , H. Sebastian Seung , Eli Shamir , Naftali Tishby, Selective Sampling Using the Query by Committee Algorithm, Machine Learning, v.28 n.2-3, p.133-168, Aug./Sept. 1997 [doi>10.1023/A:1007330508534]
• 6: C. Kerwin. The state of rulemaking in the federal government. Technical report, Transcript Panel 1, 2005.
• 7: Namhee Kwon , Eduard Hovy, Information acquisition using multiple classifications, Proceedings of the 4th international conference on Knowledge capture, October 28-31, 2007, Whistler, BC, Canada [doi>10.1145/1298406.1298427]
• 8: Namhee Kwon , Stuart W. Shulman , Eduard Hovy, Multidimensional text analysis for eRulemaking, Proceedings of the 2006 international conference on Digital government research, May 21-24, 2006, San Diego, California [doi>10.1145/1146598.1146649]
• 9: D. D. Lewis and J. Catlett. Heterogeneous Uncertainty Sampling for Supervised Learning. In Proceedings of the Eleventh International Conference on Machine Learning, pages 148--156, Rutgers University, New Brunswick, NJ, 1994. Morgan Kaufmann.
• 10: Prem Melville , Raymond J. Mooney, Diverse ensembles for active learning, Proceedings of the twenty-first international conference on Machine learning, p.74, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015385]
• 11: Ion Muslea , Steven Minton , Craig A. Knoblock, Selective Sampling with Redundant Views, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.621-626, July 30-August 03, 2000
• 12: Kishore Papineni, Why inverse document frequency?, Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001, p.1-8, June 01-07, 2001, Pittsburgh, Pennsylvania [doi>10.3115/1073336.1073340]
• 13: M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, 1980.
• 14: Stephen Purpura , Dustin Hillard, Automated classification of congressional legislation, Proceedings of the 2006 international conference on Digital government research, May 21-24, 2006, San Diego, California [doi>10.1145/1146598.1146660]
• 15: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 16: Bernhard Scholkopf , Alexander J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, 2001
• 17: H. S. Seung , M. Opper , H. Sompolinsky, Query by committee, Proceedings of the fifth annual workshop on Computational learning theory, p.287-294, July 27-29, 1992, Pittsburgh, Pennsylvania, United States [doi>10.1145/130385.130417]
• 18: S. Shulman. Perverse incentives: The case against mass e-mail campaigns. In Proceedings of the Annual Meeting of the American Political Science Association, 2008.
• 19: P. Strauss, T. Rakoff, and C. Farina. Administrative Law. 10th edition, 2003.
• 20: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 21: Hui Yang , Jamie Callan, Near-duplicate detection for eRulemaking, Proceedings of the 2005 national conference on Digital government research, May 15-18, 2005, Atlanta, Georgia
• 22: Hui Yang , Jamie Callan, Near-duplicate detection by instance-level constrained clustering, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148243]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Text analysis
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
      I.5.4 Applications
          Subjects: Text processing

  J. Computer Applications
  J.1 ADMINISTRATIVE DATA PROCESSING
      Subjects: Government

General Terms:
Algorithms, Design, Management, Theory

Keywords:
active learning, e-rulemaking, machine learning, public comment, text categorization