A common database approach for OLTP and OLAP using an in-memory column database

ABSTRACT

When SQL and the relational data model were introduced 25 years ago as a general data management concept, enterprise software migrated quickly to this new technology. It is fair to say that SQL and the various implementations of RDBMSs became the backbone of enterprise systems. In those days. we believed that business planning, transaction processing and analytics should reside in one single system. Despite the incredible improvements in computer hardware, high-speed networks, display devices and the associated software, speed and flexibility remained an issue.

The nature of RDBMSs, being organized along rows, prohibited us from providing instant analytical insight and finally led to the introduction of so-called data warehouses. This paper will question some of the fundamentals of the OLAP and OLTP separation. Based on the analysis of real customer environments and experience in some prototype implementations, a new proposal for an enterprise data management concept will be presented.

In our proposal, the participants in enterprise applications, customers, orders, accounting documents, products, employees etc. will be modeled as objects and also stored and maintained as such. Despite that, the vast majority of business functions will operate on an in memory representation of their objects. Using the relational algebra and a column-based organization of data storage will allow us to revolutionize transactional applications while providing an optimal platform for analytical data processing. The unification of OLTP and OLAP workloads on a shared architecture and the reintegration of planning activities promise significant gains in application development while simplifying enterprise systems drastically.

The latest trends in computer technology -- e.g. blade architecture, multiple CPUs per blade with multiple cores per CPU allow for a significant parallelization of application processes. The organization of data in columns supports the parallel use of cores for filtering and aggregation. Elements of application logic can be implemented as highly efficient stored procedures operating on columns. The vast increase in main memory combined with improvements in L1--, L2--, L3--caching, together with the high data compression rate column storage will allow us to support substantial data volumes on one single blade. Distributing data across multiple blades using a shared nothing approach provides further scalability.

REFERENCES

• 1: Daniel Abadi , Samuel Madden , Miguel Ferreira, Integrating compression and execution in column-oriented database systems, Proceedings of the 2006 ACM SIGMOD international conference on Management of data, June 27-29, 2006, Chicago, IL, USA [doi>10.1145/1142473.1142548]
• 2: Stefan Aulbach , Torsten Grust , Dean Jacobs , Alfons Kemper , Jan Rittinger, Multi-tenant databases for software as a service: schema-mapping techniques, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada [doi>10.1145/1376616.1376736]
• 3: Hal Berenson , Phil Bernstein , Jim Gray , Jim Melton , Elizabeth O'Neil , Patrick O'Neil, A critique of ANSI SQL isolation levels, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.1-10, May 22-25, 1995, San Jose, California, United States
• 4: A. Bog, J. Krueger, and J. Schaner. A Composite Benchmark for Online Transaction Processing and Operational Reporting. In IEEE Symposium on Advanced Management of Information for Globalized Enterprises, 2008.
• 5: P. Boncz. Monet: A Next-Generation DBMS Kernel for Query-Intensive Applications. 2002. PhD Thesis, Universiteit van Amsterdam, Amsterdam, The Netherlands.
• 6: Peter A. Boncz , Stefan Manegold , Martin L. Kersten, Database Architecture Optimized for the New Bottleneck: Memory Access, Proceedings of the 25th International Conference on Very Large Data Bases, p.54-65, September 07-10, 1999
• 7: Surajit Chaudhuri , Umeshwar Dayal, An overview of data warehousing and OLAP technology, ACM SIGMOD Record, v.26 n.1, p.65-74, March 1997 [doi>10.1145/248603.248616]
• 8: George P. Copeland , Setrag N. Khoshafian, A decomposition storage model, Proceedings of the 1985 ACM SIGMOD international conference on Management of data, p.268-279, May 1985, Austin, Texas, United States
• 9: Clark D. French, “One size fits all” database architectures do not work for DSS, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.449-450, May 22-25, 1995, San Jose, California, United States
• 10: B. Gates. Information At Your Fingertips. Keynote address, Fall/COMDEX, Las Vegas, Nevada, November 1994.
• 11: J. Gray. Tape is Dead. Disk is Tape. Flash is Disk, RAM Locality is King. Storage Guru Gong Show, Redmon, WA, 2006.
• 12: John L. Hennessy , David A. Patterson, Computer Architecture, Fourth Edition: A Quantitative Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2006
• 13: W. H. Inmon, Building the Data Warehouse,3rd Edition, John Wiley & Sons, Inc., New York, NY, 2002
• 14: G. Koch. Discovering Multi-Core: Extending the Benefits of Moore's Law. Technology@Intel, (7), 2005.
• 15: D. Majumdar. A Quick Survey of MultiVersion Concurrency Algorithms, 2007. http://simpledbm.googlecode.com/~les/mvcc-survey-1.0.pdf.
• 16: G.E. Moore. Cramming More Components Onto Integrated Circuits. Electronics, 38(8), 1965.
• 17: Vijayshankar Raman , Garret Swart , Lin Qiao , Frederick Reiss , Vijay Dialani , Donald Kossmann , Inderpal Narang , Richard Sidle, Constant-Time Query Processing, Proceedings of the 2008 IEEE 24th International Conference on Data Engineering, p.60-69, April 07-12, 2008 [doi>10.1109/ICDE.2008.4497414]
• 18: J. Schaner, A. Bog, J. Krueger, and A. Zeier. A Hybrid Row-Column OLTP Database Architecture for Operational Reporting. In Proceedings of the Second International Workshop on Business Intelligence for the Real-Time Enterprise, BIRTE 2008, in conjunction with VLDB'08, August 24, 2008, Auckland, New Zealand, 2008.
• 19: M. Stonebraker. The Case for Shared Nothing. IEEE Database Engineering Bulletin, 9(1):4--9, 1986.
• 20: Mike Stonebraker , Daniel J. Abadi , Adam Batkin , Xuedong Chen , Mitch Cherniack , Miguel Ferreira , Edmond Lau , Amerson Lin , Sam Madden , Elizabeth O'Neil , Pat O'Neil , Alex Rasin , Nga Tran , Stan Zdonik, C-store: a column-oriented DBMS, Proceedings of the 31st international conference on Very large data bases, August 30-September 02, 2005, Trondheim, Norway
• 21: M. Stonebraker , L. A. Rowe , M. Hirohama, The Implementation of POSTGRES, IEEE Transactions on Knowledge and Data Engineering, v.2 n.1, p.125-142, March 1990 [doi>10.1109/69.50912]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.0 General

General Terms:
Design, Performance

Keywords:
invited talk paper