Pragmatic text mining: minimizing human effort to quantify many issues in call logs

ABSTRACT

We discuss our experiences in analyzing customer-support issues from the unstructured free-text fields of technical-support call logs. The identification of frequent issues and their accurate quantification is essential in order to track aggregate costs broken down by issue type, to appropriately target engineering resources, and to provide the best diagnosis, support and documentation for most common issues. We present a new set of techniques for doing this efficiently on an industrial scale, without requiring manual coding of calls in the call center. Our approach involves (1) a new text clustering method to identify common and emerging issues; (2) a method to rapidly train large numbers of categorizers in a practical, interactive manner; and (3) a method to accurately quantify categories, even in the face of inaccurate classifications and training sets that necessarily cannot match the class distribution of each new month's data. We present our methodology and a tool we developed and deployed that uses these methods for tracking ongoing support issues and discovering emerging issues at HP.

REFERENCES

• 1: Arindam Banerjee , Chase Krumpelman , Joydeep Ghosh , Sugato Basu , Raymond J. Mooney, Model-based overlapping clustering, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081932]
• 2: Florian Beil , Martin Ester , Xiaowei Xu, Frequent term-based text clustering, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775110]
• 3: Deerwester, S., Dumais, S., Furnas, G, Landauer, T, and Harshman, R. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391--407, 1990.
• 4: Tom Fawcett , Peter A. Flach, A Response to Webb and Ting's On the Application of ROC Analysis to Predict Classification Performance Under Varying Class Distributions, Machine Learning, v.58 n.1, p.33-38, January 2005 [doi>10.1007/s10994-005-5256-4]
• 5: George Forman, Quantifying trends accurately despite classifier error and class imbalance, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA [doi>10.1145/1150402.1150423]
• 6: Forman, G. Counting positives accurately despite inaccurate classification. In Proc. of the 16th European Conf. on Machine Learning (ECML, Porto):564--575, 2005.
• 7: George Forman , Ira Cohen, Learning from little: comparison of classifiers given little training, Proceedings of the 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, p.161-172, September 20-24, 2004, Pisa, Italy
• 8: George Forman, An extensive empirical study of feature selection metrics for text classification, The Journal of Machine Learning Research, 3, 3/1/2003
• 9: Susan Havre , Elizabeth Hetzler , Paul Whitney , Lucy Nowell, ThemeRiver: Visualizing Thematic Changes in Large Document Collections, IEEE Transactions on Visualization and Computer Graphics, v.8 n.1, p.9-20, January 2002 [doi>10.1109/2945.981848]
• 10: Thorsten Joachims, Text Categorization with Suport Vector Machines: Learning with Many Relevant Features, Proceedings of the 10th European Conference on Machine Learning, p.137-142, April 21-23, 1998
• 11: Li, X., Wang, L., and Sung, E. Multilabel SVM active learning for image classification. In Proc. of the Int'l Conf. on Image Processing (ICIP), 4:2207--2210, 2004.
• 12: MacQueen, J. B. Some Methods for classification and Analysis of Multivariate Observations, In Proc. of 5th Berkeley Symposium on Mathematical Statistics and Probability, Univ. of California Press, 1:281--297, 1967.
• 13: Qiaozhu Mei , ChengXiang Zhai, Discovering evolutionary theme patterns from text: an exploration of temporal text mining, Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081895]
• 14: Prem Melville , Raymond J. Mooney, Diverse ensembles for active learning, Proceedings of the twenty-first international conference on Machine learning, p.74, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015385]
• 15: Monica Rogati , Yiming Yang, Resource selection for domain-specific cross-lingual IR, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009021]
• 16: Suermondt, J., Kirshenbaum, E., Forman, G., and Stinger, J. The 10-second answer: practical text clustering for topic discovery. Forthcoming. HP Labs, Tech.Rpt. HPL-2006-41.
• 17: Thearling, K. Some thoughts on the current state of data mining software applications. Workshop: Keys to the Commercial Success of Data Mining, 8th ACM SIGKDD Int'l Conf. on Knowledge Discovery in Data Mining (KDD, New York), 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.2 Types of Systems
          Subjects: Decision support (e.g., MIS)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods; Linguistic processing

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning
  I.5 PATTERN RECOGNITION
      I.5.3 Clustering
          Subjects: Algorithms
      I.5.4 Applications
          Subjects: Text processing

General Terms:
Algorithms, Design, Economics, Measurement

Keywords:
applications, log processing, quantification, supervised machine learning, text classification, text mining