Effective data mining: a data warehouse-backboned architecture

ABSTRACT

An effective Data Mining (DM) system for mining multiple-level knowledge from Data Warehouse (DW), DB and flat files of raw data is proposed. The DW represents the backbone of the proposed architecture. Intermediate, as well as final results of mining are incorporated into the DW for efficient processing of further queries. A Markov Chain mathematical model is developed for managing data dependency and consistency in the DW. An adaptive hybrid view technique is introduced to manage storage space. DM and OLAP technologies are closely integrated. The mining and OLAP kernel includes generic analysis modules for performing a wide spectrum of applications. Active data mining is adopted to support knowledge-driven business processes. Continuously gathered business data is partitioned according to application-dependent time periods. Active mining uses these partitioned data sets to discover rules and key business indicators for each time period.

REFERENCES

• 1: {1} R. Agrawal, A. Arning, T. Bollinger, M. Mehta, J. Shafer, and R. Srikant. The Quest Data Mining System. In Proceedings of the 2nd Int'l Conference on Knowledge Discovery and Data Mining, 1996.
• 2: {2} R. Agrawl and G. Psaila. Active Data Mining. In Proceedings of the 1st Int'l Conference on Knowledge Discovery and Data Mining, 1995.
• 3: {3} A. Arning, R. Agrawal, and P. Raghavan. A Linear Method for Deviataion Detection in Large Databases. In Proceedings of the 2st Int'l Conference on Knowledge Discovery and Data Mining, 1996.
• 4: Paul C. Attie , Munindar P. Singh , Amit P. Sheth , Marek Rusinkiewicz, Specifying and Enforcing Intertask Dependencies, Proceedings of the 19th International Conference on Very Large Data Bases, p.134-145, August 24-27, 1993
• 5: R. J. Bayardo, Jr. , W. Bohrer , R. Brice , A. Cichocki , J. Fowler , A. Helal , V. Kashyap , T. Ksiezyk , G. Martin , M. Nodine , M. Rashid , M. Rusinkiewicz , R. Shea , C. Unnikrishnan , A. Unruh , D. Woelk, InfoSleuth: agent-based semantic integration of information in open and dynamic environments, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.195-206, May 11-15, 1997, Tucson, Arizona, United States
• 6: Usama M. Fayyad , S. George Djorgovski , Nicholas Weir, Automating the analysis and cataloging of sky surveys, Advances in knowledge discovery and data mining, American Association for Artificial Intelligence, Menlo Park, CA, 1996
• 7: Usama Fayyad , Gregory Piatetsky-Shapiro , Padhraic Smyth, The KDD process for extracting useful knowledge from volumes of data, Communications of the ACM, v.39 n.11, p.27-34, Nov. 1996 [doi>10.1145/240455.240464]
• 8: Clark Glymour , David Madigan , Daryl Pregibon , Padhraic Smyth, Statistical inference and data mining, Communications of the ACM, v.39 n.11, p.35-41, Nov. 1996 [doi>10.1145/240455.240466]
• 9: {9} J. Hammer, H. Garcia-Molina, J. Widom, W. Labio, and Y. Zhuge. The Stanford Data Warehousing Project. "IEEE Data Engineering Bulletin", 18(2):41-48, June 1995.
• 10: Jiawei Han, Towards on-line analytical mining in large databases, ACM SIGMOD Record, v.27 n.1, p.97-107, March 1998 [doi>10.1145/273244.273273]
• 11: {11} A. Helal, Y. Taha, K. Ahmed, and M. Nagy. Modeling Data Dependency and consistency in Data Warehousing. In Proceedings of the 7th Intl. Conference on Computer Theory and Applications, Alexandria, Egypt, September 1997.
• 12: Richard Hull , Gang Zhou, A framework for supporting data integration using the materialized and virtual approaches, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.481-492, June 04-06, 1996, Montreal, Quebec, Canada
• 13: Christopher J. Matheus , Gregory Piatetsky-Shapiro , Dwight McNeill, Selecting and reporting what is interesting, Advances in knowledge discovery and data mining, American Association for Artificial Intelligence, Menlo Park, CA, 1996
• 14: {14} S. Ross. Introduction to Probability Models . Acadimic Press Inc., 1985.
• 15: Marek Rusinkiewicz , Amit Sheth , George Karabatis, Specifying Interdatabase Dependencies in a Multidatabase Environment, Computer, v.24 n.12, p.46-53, December 1991 [doi>10.1109/2.116888]
• 16: {16} Y. Taha. Managing Data, Task Dependency and Consistency in Data Warehousing . PhD thesis, Alexandria University, June 1997.
• 17: {17} Y. Taha, A. Helal, and K. Ahmed. A Stochastic Consistency Model for Data Warehousing. In Proceedings of the 3rd American Conference on Information Systems , Indianapolis, Indiana, USA, August 1997.
• 18: {18} Y. Taha, A. Helal, and K. Ahmed. Data Warehousing: Usage, Architecture, and Research Issues. ISMM Microcomputer applications journal, 16(2), 1997.
• 19: {19} J. Wiener, H. Gupta, W. Labio, Y. Zhuge, H. Garcia-Molina, and J. Widom. A System Prototype for Warehouse View Maintenance. In Proceedings of the ACM Workshop on Materialized Views: Techniques and Applications, pages 26-33, Montreal, Canada, June 1996.
• 20: {20} G. Zhou, R. Hull, R. King, and J. Franchitti. Data Integration and Warehousing Using H2O. IEEE Data Engineering Bulletin , 18(2):29-40, June 1995.
• 21: Yue Zhuge , Héctor García-Molina , Joachim Hammer , Jennifer Widom, View maintenance in a warehousing environment, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.316-327, May 22-25, 1995, San Jose, California, United States
• 22: Yue Zhuge , Hector Garcia-Molina , Janet L. Wiener, The Strobe algorithms for multi-source warehouse consistency, Proceedings of the fourth international conference on on Parallel and distributed information systems, p.146-157, December 18-20, 1996, Miami Beach, Florida, United States

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Markov processes

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.1 Logical Design
          Subjects: Data models
      H.2.7 Database Administration
          Subjects: Data warehouse and repository

General Terms:
Algorithms, Design, Experimentation, Measurement, Performance, Theory