Peta-scale data warehousing at Yahoo!

ABSTRACT

Insights based on detailed data on consumer behavior, product performance and marketplace behavior are driving innovation and competition in the internet space. We introduce Everest, a SQL-compliant data warehousing engine, based on a column architecture that we have built and deployed at Yahoo!. In contrast to commercially available engines, this massively parallel engine, based on commodity hardware, offers scale, flexibility, specialized analytic operations, and lower administrative & hardware costs. In this paper, we describe the business motivation and the software and deployment architecture of Everest. The engine is in production at Yahoo! since 2007 and currently manages over six petabytes of data.

REFERENCES

• 1: Alon Halevy , Peter Norvig , Fernando Pereira, The Unreasonable Effectiveness of Data, IEEE Intelligent Systems, v.24 n.2, p.8-12, March 2009 [doi>10.1109/MIS.2009.36]
• 2: Prahalad, C.K., and Krishnan, M.S., The New Age of Innovation: Driving Cocreated Value Through Global Network. The McGraw Hill Companies, 2008
• 3: http://www.postgresql.org/
• 4: Daniel J. Abadi , Samuel Madden, Query execution in column-oriented database systems, Massachusetts Institute of Technology, Cambridge, MA, 2008
• 5: David DeWitt , Jim Gray, Parallel database systems: the future of high performance database systems, Communications of the ACM, v.35 n.6, p.85-98, June 1992 [doi>10.1145/129888.129894]
• 6: Yihong Zhao , Prasad M. Deshpande , Jeffrey F. Naughton , Amit Shukla, Simultaneous optimization and evaluation of multiple dimensional queries, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.271-282, June 01-04, 1998, Seattle, Washington, United States
• 7: Kevin Lim , Parthasarathy Ranganathan , Jichuan Chang , Chandrakant Patel , Trevor Mudge , Steven Reinhardt, Understanding and Designing New Server Architectures for Emerging Warehouse-Computing Environments, Proceedings of the 35th International Symposium on Computer Architecture, p.315-326, June 21-25, 2008
• 8: Hong, W., Parallel Query Processing Using Shared Memory Multiprocessors and Disk Arrays, PhD Dissertation, EECS Department, University of California, Berkeley, 1993
• 9: Waqar Hasan , Gio Wiederhold, Optimization of SQL Queries for Parallel Machines, Springer-Verlag New York, Inc., Secaucus, NJ, 1996
• 10: Danette Chimenti , Ruben Gamboa , Ravi Krishnamurthy, Abstract machine for LDL, Proceedings of the 2nd international conference on extending database technology: Advances in Database Technology, p.153-168, March 1990, Venice, Italy

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Distributed systems

General Terms:
Algorithms, Languages, Performance, Reliability, Standardization

Keywords:
analytics, business intelligence, column database, column storage, data warehousing, mpp database, vector query processing