Maintenance policy selection in heterogeneous data warehouse environments: a heuristics-based approach

ABSTRACT

This work addresses data warehouse maintenance, i.e. how changes to autonomous, heterogeneous, and distributed sources should be detected and propagated to a warehouse. The research community has mainly addressed issues relating to the internal operation of data warehouse servers. Work related to data warehouse maintenance has received less attention and only a limited set of maintenance alternatives are considered while ignoring the autonomy and heterogeneity of sources.In this paper, we extend work on single source view maintenance to views with multiple heterogeneous sources. We present a tool (PAM) which allows for comparison of a large number of relevant maintenance policies under different configurations. Based on such analysis and previous studies we propose a set of heuristics to guide in policy selection. The quality of these heuristics is evaluated empirically using a test-bed developed for this purpose. This is done for a number of different criteria and for different data sources and computer systems. The performance gained using the policy selected through the heuristics is compared with the performance of all identified policies. Based on these experiments we claim that heuristic-based selections are good.

REFERENCES

• 1: D. Agrawal , A. El Abbadi , A. Singh , T. Yurek, Efficient view maintenance at data warehouses, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.417-427, May 11-15, 1997, Tucson, Arizona, United States
• 2: Rakesh Agrawal , Ashish Gupta , Sunita Sarawagi, Modeling Multidimensional Databases, Proceedings of the Thirteenth International Conference on Data Engineering, p.232-243, April 07-11, 1997
• 3: M. Akhtar Ali , Norman W. Paton , Alvaro A. A. Fernandes, An Experimental Performance Evaluation of Incremental Materialized View Maintenance in Object Databases, Proceedings of the Third International Conference on Data Warehousing and Knowledge Discovery, p.240-253, September 05-07, 2001
• 4: Latha S. Colby , Akira Kawaguchi , Daniel F. Lieuwen , Inderpal Singh Mumick , Kenneth A. Ross, Supporting multiple view maintenance policies, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.405-416, May 11-15, 1997, Tucson, Arizona, United States
• 5: Henrik Engström , Sharma Chakravarthy , Brian Lings, A User-centric View of Data Warehouse Maintenance Issues, Proceedings of the 17th British National Conferenc on Databases: Advances in Databases, p.68-80, July 03-05, 2000
• 6: H. Engström, S. Chakravarthy, and B. Lings. Data integration in heterogeneous environments: Multi-source policies, cost model, and implementation. Technical report, University of Skövde, Sweden, 2002.
• 7: Henrik Engström , Sharma Chakravarthy , Brian Lings, Implementation and Comparative Evaluation of Maintenance Policies in a Data Warehouse Environment, Proceedings of the 19th British National Conference on Databases: Advances in Databases, p.90-102, July 17-19, 2002
• 8: Henrik Engström , Sharma Chakravarthy , Brian Lings, A Systematic Approach to Selecting Maintenance Policies in a Data Warehouse Environment, Proceedings of the 8th International Conference on Extending Database Technology: Advances in Database Technology, p.317-335, March 25-27, 2002
• 9: Ashish Gupta , H. V. Jagadish , Inderpal Singh Mumick, Data Integration using Self-Maintainable Views, Proceedings of the 5th International Conference on Extending Database Technology: Advances in Database Technology, p.140-144, March 25-29, 1996
• 10: A. Gupta and I. S. Mumick. Maintenance of materialized views: Problems, techniques, and applications. IEEE Data Engineering Bulletin, 18(2):3--18, 1995.
• 11: J. Hammer, H. Garcia-Molina, J. Widom, W. Labio, and Y. Zhuge. The Stanford data warehousing project. IEEE Data Engineering Bulletin, 18(2):41--48, 1995.
• 12: Eric N. Hanson, A performance analysis of view materialization strategies, Proceedings of the 1987 ACM SIGMOD international conference on Management of data, p.440-453, May 27-29, 1987, San Francisco, California, United States
• 13: Venky Harinarayan , Anand Rajaraman , Jeffrey D. Ullman, Implementing data cubes efficiently, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.205-216, June 04-06, 1996, Montreal, Quebec, Canada
• 14: Richard Hull , Gang Zhou, A framework for supporting data integration using the materialized and virtual approaches, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.481-492, June 04-06, 1996, Montreal, Quebec, Canada
• 15: Yannis Kotidis , Nick Roussopoulos, An alternative storage organization for ROLAP aggregate views based on cubetrees, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.249-258, June 01-04, 1998, Seattle, Washington, United States
• 16: M. Lee and J. Hammer. Speeding up warehouse physical design using a randomized algorithm. DMDW Workshop, 1999.
• 17: Dallan Quass , Ashish Gupta , Inderpal Singh Mumick , Jennifer Widom, Making views self-maintainable for data warehousing, Proceedings of the fourth international conference on on Parallel and distributed information systems, p.158-169, December 18-20, 1996, Miami Beach, Florida, United States
• 18: Arie Segev , Weiping Fang, Optimal update policies for distributed materialized views, Management Science, v.37 n.7, p.851-870, July 1991 [doi>10.1287/mnsc.37.7.851]
• 19: A. Segev , J. Park, Updating Distributed Materialized Views, IEEE Transactions on Knowledge and Data Engineering, v.1 n.2, p.173-184, June 1989 [doi>10.1109/69.87958]
• 20: Jaideep Srivastava , Doron Rotem, Analytical modeling of materialized view maintenance, Proceedings of the seventh ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, p.126-134, March 1988, Austin, Texas, United States [doi>10.1145/308386.308425]
• 21: Dimitri Theodoratos , Timos K. Sellis, Data Warehouse Configuration, Proceedings of the 23rd International Conference on Very Large Data Bases, p.126-135, August 25-29, 1997
• 22: Hui Wang , Maria Orlowska , Weifa Liang, Efficient refreshment of materialized views with multiple sources, Proceedings of the eighth international conference on Information and knowledge management, p.375-382, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320030]
• 23: Xin Zhang , Lingli Ding , Elke A. Rundensteiner, PVM: Parallel View Maintenance under Concurrent Data Updates of Distributed Sources, Proceedings of the Third International Conference on Data Warehousing and Knowledge Discovery, p.230-239, September 05-07, 2001
• 24: Yue Zhuge , Hector Garcia-Molina, Incremental maintenance of consistent data warehouses, 1999
• 25: Yue Zhuge , Héctor García-Molina , Joachim Hammer , Jennifer Widom, View maintenance in a warehousing environment, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.316-327, May 22-25, 1995, San Jose, California, United States
• 26: Yue Zhuge , Hector Garcia-Molina , Janet L. Wiener, The Strobe algorithms for multi-source warehouse consistency, Proceedings of the fourth international conference on on Parallel and distributed information systems, p.146-157, December 18-20, 1996, Miami Beach, Florida, United States

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems

General Terms:
Performance

Keywords:
consistency, data integration, data warehouse, policies, view maintenance