Mining for patterns in contradictory data

ABSTRACT

Information integration is often faced with the problem that different data sources represent the same set of the real-world objects, but give conflicting values for specific properties of these objects. Within this paper we present a model of such conflicts and describe an algorithm for efficiently detecting patterns of conflicts in a pair of overlapping data sources. The contradiction patterns we can find are a special kind of association rules, describing regularities in conflicts occurring together with certain attribute values, paris of attribute values, or with other conflicts. Therefore, we adapt existing association rule mining algorithms for mining contradiction patterns. Such patterns are an important tool for human experts that try to find and resolve problems in data quality using domain knowledge. We present the results of applying our method on a real world data set from the life science domain and show how it helps to generate clean data for integrated data warehouses.

REFERENCES

• 1: Felix Naumann, Quality-driven query answering for integrated information systems, Springer-Verlag New York, Inc., New York, NY, 2002
• 2: Jack Olson, Data Quality: The Accuracy Dimension, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2002
• 3: Rother, K., Müller, H., Trissl, S., Koch, I., Steinke, T., Preissner, R., Frömmel, C., Leser, U., COLUMBA: Multidimensional Data Integration of Protein Annotations. International Workshop on Data Integration in Life Sciences (DILS 2004), Leipzig, Germany
• 4: Benson, D. A., Karsch-Mizrachi, I., Lipman, D. J., Ostell, J., Wheeler, D. L. GenBank. Nucleic Acids Res., Vol. 31(1) (2003) 23--37
• 5: Kulikova, T., Aldebert, P., Althorpe, N., Baker, W., Bates, K., Browne, P., van den Broek, A., Cochrane, G., Duggan, K., Eberhardt, R., Faruque, N., Garcia-Pastor, M., Harte, N., Kanz, C., Leinonen, R., Lin, Q., Lombard, V., Lopez, R., Mancuso, R., McHale, M., Nardone, F., Silventoinen, V., Stoehr, P., Stoesser, G., Tuli, M. A., Tzouvara, K., Vaughan, R., Wu, D., Zhu, W., Apweiler. R., The EMBL Nucleotide Sequence Database. Nucl. Acids. Res. 2004 32:D27-D30.
• 6: Tateno, Y., Imanishi, T., Miyazaki, S., Fukami-Kobayashi, K., Saitou, N., Sugawara, H., Gojobori, T., DNA Data Bank of Japan (DDBJ) for genome scale research in life science. Nucl. Acids. Res. 2002 30:27--30
• 7: Bernstein, F. C., Koetzle, T. F., Willliams, G. J. B., Meyer, E. F. Jr., Brice, M. D., Rodgers, J. R., Kennard, O., Shimanouchi, T., Tasumi, M., The Protein Data Bank: a computer-based archival file for macromolecular structures. J. Mol. Biol., Vol. 112 (1977) 535--542
• 8: Bhat, T. N., Bourne, P., Feng, Z., Gilliland, G., Jain, S., Ravichandran, V., Schneider, B., Schneider, K., Thanki, N., Weissig, H., Westbrook, J., Berman, H. M., The PDB data uniformity project. Nucleic Acid Research, Vol. 29(1) (2001) 214--218
• 9: Boutselakis, H., Dimitropoulos, D., Fillon, J., Golovin, A., Henrick, K., Hussain, A., Ionides, J., John, M., Keller, P. A., Krissinel, E., McNeil, P., Naim, A., Newman, R., Oldfield, T., Pineda, J., Rachedi, A., Copeland, J., Sitnov, A., Sobhany, S., Suarez-Urunea, A., Swaminathan, J., Tagari, M., Tate, J., Tromm, S., Velankar, S., Vranken, W.; E-MSD: the European Bioinformatics Institute Macromolecular Structure Database. Nucleic Acid Research, Vol. 31(1) (2003) 458--462
• 10: Tamraparni Dasu , Theodore Johnson, Exploratory Data Mining and Data Cleaning, John Wiley & Sons, Inc., New York, NY, 2003
• 11: Naumann, F., Haeussler, M., Declarative Data Merging with Conflict Resolution. Proceedings of the International Conference on Information Quality (IQ 2002), Cambridge, MA, 2002
• 12: Müller, H., Naumann, F., Freytag, J. C., Data Quality in Genome Databases. Proceedings of the International Conference on Information Quality (IQ 2003), Cambridge, MA, 2003
• 13: Hui Xiong , Pang-Ning Tan , Vipin Kumar, Mining Strong Affinity Association Patterns in Data Sets with Skewed Support Distribution, Proceedings of the Third IEEE International Conference on Data Mining, p.387, November 19-22, 2003
• 14: Rakesh Agrawal , Ramakrishnan Srikant, Fast Algorithms for Mining Association Rules in Large Databases, Proceedings of the 20th International Conference on Very Large Data Bases, p.487-499, September 12-15, 1994
• 15: Ramakrishnan Srikant , Rakesh Agrawal, Mining quantitative association rules in large relational tables, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.1-12, June 04-06, 1996, Montreal, Quebec, Canada