Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

An important step in building effective predictive models applies one or more sampling techniques. Traditional sampling techniques include random, stratified, systemic, and clustered. The problem with these techniques is that they focus on the class attribute, rather than the non-class attributes. For example, if a test instance's nearest neighbor is from the opposite class of the training set, then it seems doomed to misclassification. To illustrate this problem, this paper conducts 20 experiments on five different NASA defect datasets (CM1, JM1, KC1, KC2, PC1) using two different learners (J48 and Naïve Bayes). Each data set is divided into 3 groups, a training set, and "nice/nasty" neighbor test sets. Using a nearest neighbor approach, "Nice neighbors" consist of those test instances closest to class training instances. "Nasty neighbors" are closest to opposite class training instances. The "Nice" experiments average 94 percent accuracy and the "Nasty" experiments average 20 percent accuracy. Based on these results a new nearest neighbor sampling technique is proposed.

REFERENCES

• 1: Boetticher, G., Data Mining Software Tools (2005), CSCI5833 Data Mining Tools and Techniques, Department of Computer Science, University of Houston - Clear Lake, Houston, Texas. Available at: <u>http://nas.cl.uh.edu/boetticher/CSCI5931%20Data%20Mining.html</u>
• 2: Jorgensen, M., "A review of studies on Expert Estimation of Software Development Effort," Journal of Systems and Software, 70 (1--2), Pp. 37--60, 2004.
• 3: KDD Nuggets Website, Polls: Data Mining Tools You Regularly Use, Knowledge Discovery and Data Mining Poll, <u>http://www.kdnuggets.com/polls/data mining tools 2002 june2.htm</u>
• 4: Khoshgoftaar, T. M., and E. B. Allen, "Model software quality with classification trees," in Recent Advances in Reliability and Quality Engineering, H. Pham, Ed. 2001, pp. 247--270, World Scientific.
• 5: Menzies, Tim, Personal Conversation, February 2, 2005.
• 6: Menzies, T., Raffo D., Setamanit, S., DiStefano, J., Chapman, R., Why Mine Repositories, Submitted to: Transactions on Software Engineering, 2005.
• 7: Adam A. Porter , Richard W. Selby, Empirically Guided Software Development Using Metric-Based Classification Trees, IEEE Software, v.7 n.2, p.46-54, March 1990 [doi>10.1109/52.50773]
• 8: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 9: Rish, I., An empirical study of the naive Bayes classifier, T. J. Watson Center, IJCAI-01 Workshop on Empirical Methods in Artificial Intelligence, Seattle, 2001.
• 10: Shirabad, J. S., and Menzies, T. J. (2005) The PROMISE Repository of Software Engineering Databases School of Information Technology and Engineering, University of Ottawa, Canada Available: <u>http://promise.site.uottawa.ca/SERepository</u>
• 11: Krishnamoorthy Srinivasan , Douglas Fisher, Machine Learning Approaches to Estimating Software Development Effort, IEEE Transactions on Software Engineering, v.21 n.2, p.126-137, February 1995 [doi>10.1109/32.345828]
• 12: Jeff Tian , Marvin V. Zelkowitz, Complexity Measure Evaluation and Selection, IEEE Transactions on Software Engineering, v.21 n.8, p.641-650, August 1995 [doi>10.1109/32.403788]
• 13: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Knowledge acquisition

General Terms:
Experimentation, Measurement

Keywords:
NASA data repository, decision trees, defect prediction, empirical software engineering, nearest neighbor analysis