Stream warehousing with DataDepot

ABSTRACT

We describe DataDepot, a tool for generating warehouses from streaming data feeds, such as network-traffic traces, router alerts, financial tickers, transaction logs, and so on. DataDepot is a streaming data warehouse designed to automate the ingestion of streaming data from a wide variety of sources and to maintain complex materialized views over these sources. As a streaming warehouse, DataDepot is similar to Data Stream Management Systems (DSMSs) with its emphasis on temporal data, best-effort consistency, and real-time response. However, as a data warehouse, DataDepot is designed to store tens to hundreds of terabytes of historical data, allow time windows measured in years or decades, and allow both real-time queries on recent data and deep analyses on historical data. In this paper we discuss the DataDepot architecture, with an emphasis on several of its novel and critical features. DataDepot is currently being used for five very large warehousing projects within AT&T; one of these warehouses ingests 500 Mbytes per minute (and is growing). We use these installations to illustrate streaming warehouse use and behavior, and design choices made in developing DataDepot. We conclude with a discussion of DataDepot applications and the efficacy of some optimizations.

REFERENCES

• 1: Paul G. Brown , Peter J. Haas, Techniques for Warehousing of Sample Data, Proceedings of the 22nd International Conference on Data Engineering, p.6, April 03-07, 2006 [doi>10.1109/ICDE.2006.157]
• 2: Graham Cormode , Lukasz Golab , Korn Flip , Andrew McGregor , Divesh Srivastava , Xi Zhang, Estimating the confidence of conditional functional dependencies, Proceedings of the 35th SIGMOD international conference on Management of data, June 29-July 02, 2009, Providence, Rhode Island, USA [doi>10.1145/1559845.1559895]
• 3: Wenfei Fan , Floris Geerts , Xibei Jia , Anastasios Kementsietsidis, Conditional functional dependencies for capturing data inconsistencies, ACM Transactions on Database Systems (TODS), v.33 n.2, p.1-48, June 2008 [doi>10.1145/1366102.1366103]
• 4: Nathan Folkert , Abhinav Gupta , Andrew Witkowski , Sankar Subramanian , Srikanth Bellamkonda , Shrikanth Shankar , Tolga Bozkaya , Lei Sheng, Optimizing refresh of a set of materialized views, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway
• 5: Lukasz Golab , Theodore Johnson , Vladislav Shkapenyuk, Scheduling Updates in a Real-Time Stream Warehouse, Proceedings of the 2009 IEEE International Conference on Data Engineering, p.1207-1210, March 29-April 02, 2009 [doi>10.1109/ICDE.2009.202]
• 6: Lukasz Golab , Howard Karloff , Flip Korn , Divesh Srivastava , Bei Yu, On generating near-optimal tableaux for conditional functional dependencies, Proceedings of the VLDB Endowment, v.1 n.1, August 2008 [doi>10.1145/1453856.1453900]
• 7: Rick Greer, Daytona and the fourth-generation language Cymbal, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.525-526, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 8: Theodore Johnson , Damianos Chatziantoniou, Extending complex ad-hoc OLAP, Proceedings of the eighth international conference on Information and knowledge management, p.170-179, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.319975]
• 9: Wolfgang Lehner , Roberta Cochrane , Hamid Pirahesh , Markos Zaharioudakis, fAST Refresh using Mass Query Optimization, Proceedings of the 17th International Conference on Data Engineering, p.391-398, April 02-06, 2001
• 10: N. Polyzotis, S. Skiadopoulos, P. Vassiliadis, A. Simitsis, and N.-E. Frantzell. Supporting Streaming Updates in an Active Data Warehouse. ICDE 2007, 476--485.
• 11: Dallan Quass , Jennifer Widom, On-line warehouse view maintenance, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.393-404, May 11-15, 1997, Tucson, Arizona, United States
• 12: Peter A. Tucker , David Maier, Punctuated data streams, OGI School of Science & Engineering, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.7 Database Administration
          Subjects: Data warehouse and repository

General Terms:
Algorithms, Design

Keywords:
data stream warehousing, real-time data warehousing