Investigating data quality problems in the PSP

ABSTRACT

The Personal Software Process (PSP) is used by software engineers to gather and analyze data about their work. Published studies typically use data collected using the PSP to draw quantitative conclusions about its impact upon programmer behavior and product quality. However, our experience using PSP in both industrial and academic settings revealed problems both in collection of data and its later analysis. We hypothesized that these two kinds of data quality problems could make a significant impact upon the value of PSP measures. To test this hypothesis, we built a tool to automate the PSP and then examined 89 projects completed by ten subjects using the PSP manually in an educational setting. We discovered 1539 primary errors and categorized them by type, subtype, severity, and age. To examine the collection problem we looked at the 90 errors that represented impossible combinations of data and at other less concrete anomalies in Time Recording Logs and Defect Recording Logs. To examine the analysis problem we developed a rule set, corrected the errors as far as possible, and compared the original and corrected data. This resulted in significant differences for measures such as yield and the cost-performance ratio, confirming our hypothesis. Our results raise questions about the accuracy of manually collected and analyzed PSP data, indicate that integrated tool support may be required for high quality PSP data analysis, and suggest that external measures should be used when attempting to evaluate the impact of the PSP upon programmer behavior and product quality.

REFERENCES

• 1: Anna Ch. Ceberio-Verghese, Personal Software Process: A User's Perspective, Proceedings of the 9th Conference on Software Engineering Education, p.52, April 21-24, 1996
• 2: Progress Software (Data Language Corporation). Information is available at: www.progress.comlcore/develop.htm.
• 3: George A. Ferguson and Yoshio Takane. Statistical Analysis In Psychology And Education. McGraw-Hill Book Company, 6th edition, 1989.
• 4: Pat Ferguson , Watts S. Humphrey , Soheil Khajenoori , Susan Macke , Annette Matvya, Results of Applying the Personal Software Process, Computer, v.30 n.5, p.24-31, May 1997 [doi>10.1109/2.589907]
• 5: Tom Gilb , Dorothy Graham , Susannah Finzi, Software Inspection, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1993
• 6: Will Hayes and James W. Over. The Personal Software Process (PSP): An empirical study of the impact of PSP on individual engineers. Technical Report CMU/SEI-97-TR-001, Software Eng. Inst., Pittsburgh, 1997.
• 7: Joel Henry. Personal software process studio. http://wwwcs.etsu.edu/softeng/psp/, 1997.
• 8: Watts S. Humphrey, A Discipline for Software Engineering, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 9: Watts S. Humphrey, Using A Defined and Measured Personal Software Process, IEEE Software, v.13 n.3, p.77-88, May 1996 [doi>10.1109/52.493023]
• 10: Watts S. Humphrey, Introduction to the personal software process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 11: CORPORATE Carnegie Mellon University , Mark C. Paulk , Charles V. Weber , Bill Curtis , Mary Beth Chrissis, The capability maturity model: guidelines for improving the software process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 12: James E. Tomayko, Carnegie Mellon's Software Development Studio: A Five Year Retrospectiv, Proceedings of the 9th Conference on Software Engineering Education, p.119, April 21-24, 1996

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software process

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA); Software process models (e.g., CMM, ISO, PSP)

General Terms:
Design, Measurement, Performance, Reliability, Theory

Keywords:
automated process support, defects, empirical software engineering, measurement dysfunction, personal software process