Mining concepts from code with probabilistic topic models

ABSTRACT

We develop and apply statistical topic models to software as a means of extracting concepts from source code. The effectiveness of the technique is demonstrated on 1,555 projects from SourceForge and Apache consisting of 113,000 files and 19 million lines of code. In addition to providing an automated, unsupervised, solution to the problem of summarizing program functionality, the approach provides a probabilistic framework with which to analyze and visualize source file similarity. Finally, we introduce an information-theoretic approach for computing tangling and scattering of extracted concepts, and present preliminary results

REFERENCES

• 1: Sushil Bajracharya , Trung Ngo , Erik Linstead , Yimeng Dou , Paul Rigor , Pierre Baldi , Cristina Lopes, Sourcerer: a search engine for open source code supporting structure-based search, Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications, October 22-26, 2006, Portland, Oregon, USA [doi>10.1145/1176617.1176671]
• 2: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003
• 3: S. Deerwester, S. Dumais, T. Landauer, G. Furnas, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6):391--407, 1990.
• 4: G. Kiczales, J. Lamping, A. Menhdhekar, C. Maeda, C. Lopes, J. Loingtier, and J. Irwin. Aspect-oriented programming. In M. Akşit and S. Matsuoka, editors, Proceedings European Conference on Object-Oriented Programming, volume 1241, pages 220--242. Springer-Verlag, Berlin, Heidelberg, and New York, 1997.
• 5: Adrian Kuhn , Stéphane Ducasse , Tudor Gírba, Semantic clustering: Identifying topics in source code, Information and Software Technology, v.49 n.3, p.230-243, March, 2007 [doi>10.1016/j.infsof.2006.10.017]
• 6: Erik Linstead , Paul Rigor , Sushil Bajracharya , Cristina Lopes , Pierre Baldi, Mining Eclipse Developer Contributions via Author-Topic Models, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.30, May 20-26, 2007 [doi>10.1109/MSR.2007.20]
• 7: Andrian Marcus , Andrey Sergeyev , Vaclav Rajlich , Jonathan I. Maletic, An Information Retrieval Approach to Concept Location in Source Code, Proceedings of the 11th Working Conference on Reverse Engineering, p.214-223, November 08-12, 2004
• 8: David J. Newman , Sharon Block, Probabilistic topic decomposition of an eighteenth-century American newspaper, Journal of the American Society for Information Science and Technology, v.57 n.6, p.753-767, April 2006 [doi>10.1002/asi.v57:6]
• 9: Secil Ugurel , Robert Krovetz , C. Lee Giles, What's the code?: automatic classification of source code archives, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775141]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.m Miscellaneous

General Terms:
Algorithms, Experimentation

Keywords:
mining software, program understanding, topic models