Issues and effort in integrating data from heterogeneous software repositories and corporate databases

ABSTRACT

Software repositories and corporate databases capture different fragments of a project's history. Software cockpits integrate the data from these repositories and databases to provide a holistic view of the project and the capability to drill-down and analyze details. By incorporating existing data, the cockpit can be used effectively from the first day it is introduced. In this paper we describe our findings from integrating several repositories and databases for a large, distributed project. We highlight common issues in data integration, report on the resulting effort for the development of software cockpits, and share our lessons learned from this data integration project.

REFERENCES

• 1: Donald P. Ballou , Giri Kumar Tayi, Enhancing data quality in data warehouse environments, Communications of the ACM, v.42 n.1, p.73-78, Jan. 1999 [doi>10.1145/291469.291471]
• 2: Mauricio A. Hernández , Salvatore J. Stolfo, Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem, Data Mining and Knowledge Discovery, v.2 n.1, p.9-37, January 1998 [doi>10.1023/A:1009761603038]
• 3: W. H. Inmon, Building the Data Warehouse,3rd Edition, John Wiley & Sons, Inc., New York, NY, 2002
• 4: Won Kim , Byoung-Ju Choi , Eui-Kyeong Hong , Soo-Kyung Kim , Doheon Lee, A Taxonomy of Dirty Data, Data Mining and Knowledge Discovery, v.7 n.1, p.81-99, January 2003 [doi>10.1023/A:1021564703268]
• 5: Ralph Kimball , Joe Caserta, The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleanin, John Wiley & Sons, 2004
• 6: Barbara Kitchenham , Cat Kutay , Ross Jeffery , Colin Connaughton, Lessons learnt from the analysis of large-scale corporate databases, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134347]
• 7: Ioannis Kopanas , Nikolaos M. Avouris , Sophia Daskalaki, The Role of Domain Knowledge in a Large Scale Data Mining Project, Proceedings of the Second Hellenic Conference on AI: Methods and Applications of Artificial Intelligence, p.288-299, April 11-12, 2002
• 8: Münch J., Heidrich, J. 2004. Software Project Control Centers: Concepts and Approaches. Journal of Systems and Software, Feb. 2004, 70(1-2): 3--19.
• 9: Andrew I. Schein , Alexandrin Popescul , Lyle H. Ungar , David M. Pennock, Methods and metrics for cold-start recommendations, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564421]
• 10: Ziegler, P., Dittrich, K. R. 2004. Three Decades of Data Integration - All Problems Solved? In Jacquart, R. (ed.) 18th IFIP World Computer Congress (WCC 2004), vol. 12, 3--12, Kluwer, Toulouse.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics

General Terms:
Management, Measurement

Keywords:
data integration, software cockpit, software repository