Architecture of the internet archive

ABSTRACT

The Internet Archive is a live production system supporting close to a petabyte of data and delivering an average of 2.3Gb/sec of data to Internet users. We describe the architecture of this system with an emphasis on its robustness and how it is managed by a very small team of systems personnel. Notably, the current system does not employ a cache. We analyze the reasons for this decision and show that an effective cache could not be built until now. However, new solid state disk technology may offer promising new cache implementations.

REFERENCES

• 1: Apache Software Foundation. Hadoop Core, 2008. http://hadoop.apache.org/core/.
• 2: Bibliotheca Alexandria, 2009. http://www.bibalex.org.
• 3: D. Borthakur. The Hadoop Distributed File System: Architecture and Design. The Apache Software Foundation, 2007.
• 4: M. Burner and B. Khale. WWW Archive File Format Specification, 2002. http://web.archive.org/web/20021002080721/pages.alexa.com/company/arcformat.html.
• 5: Ali R. Butt , Chris Gniady , Y. Charlie Hu, The performance impact of kernel prefetching on buffer cache replacement algorithms, ACM SIGMETRICS Performance Evaluation Review, v.33 n.1, June 2005
• 6: Carnegie Mellon University Libraries. Frequently Asked Questions About the Million Book Project, 2008. http://www.library.cmu.edu/Libraries/MBP_FAQ.html.
• 7: T.-F. Chen , J.-L. Baer, A performance study of software and hardware data prefetching schemes, Proceedings of the 21st annual international symposium on Computer architecture, p.223-232, April 18-21, 1994, Chicago, Illinois, United States
• 8: Dennis Colarelli , Dirk Grunwald, Massive arrays of idle disks for storage archives, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-11, November 16, 2002, Baltimore, Maryland
• 9: B. D. Davison. A survey of proxy cache evaluation techniques. In WCW99: Proceedings of the Fourth International Web Caching Workshop, pages 67--77, 1999.
• 10: Denice Deatrich , Simon Liu , Chris Payne , Réda Tafirout , Rodney Walker , Andrew Wong , Michel Vetterli, Managing Petabyte-Scale Storage for the ATLAS Tier-1 Centre at TRIUMF, Proceedings of the 2008 22nd International Symposium on High Performance Computing Systems and Applications, p.167-171, June 09-11, 2008 [doi>10.1109/HPCS.2008.27]
• 11: Brad Fitzpatrick, Distributed caching with memcached, Linux Journal, v.2004 n.124, p.5, August 2004
• 12: C. Gaspar. Deploying Nagios in a Large Enterprise Environment. In LISA. USENIX, 2007.
• 13: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, ACM SIGOPS Operating Systems Review, v.37 n.5, December 2003
• 14: Jim Gray , Bob Fitzgerald, Flash Disk Opportunity for Server Applications, Queue, v.6 n.4, July/August 2008 [doi>10.1145/1413254.1413261]
• 15: Rules of Thumb in Data Engineering, Proceedings of the 16th International Conference on Data Engineering, p.3, February 28-March 03, 2000
• 16: W. Hou and O. Okogbaa. Reliability and availability cost design tradeoffs for HA systems. Reliability and Maintainability Symposium, 2005. Proceedings. Annual, pages 433--438, 24--27, 2005.
• 17: T. Kelly. Priority depth (generalized stack distance) implementation in ANSI C, 2000. http://ai.eecs.umich.edu/œtpkelly/papers/.
• 18: T. Kelly and D. Reeves. Optimal web cache sizing: scalable methods for exact solutions. Computer Communications, 24(2):163--173, 2001.
• 19: Philippe Kruchten, The 4+1 View Model of Architecture, IEEE Software, v.12 n.6, p.42-50, November 1995 [doi>10.1109/52.469759]
• 20: N. Markatchev , C. Williamson, WebTraff: A GUI for Web Proxy Cache Workload Modeling and Analysis, Proceedings of the 10th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, p.356, October 11-16, 2002
• 21: R. L. Mattson, J. Gecsei, D. R. Slutz, and I. L. Traiger. Evaluation techniques for storage hierarchies. IBM Systems Journal, 9(2):78, 1970.
• 22: Eduardo Pinheiro , Ricardo Bianchini, Energy conservation techniques for disk array-based servers, Proceedings of the 18th annual international conference on Supercomputing, June 26-July 01, 2004, Malo, France [doi>10.1145/1006209.1006220]
• 23: Eduardo Pinheiro , Wolf-Dietrich Weber , Luiz André Barroso, Failure trends in a large disk drive population, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.2-2, February 13-16, 2007, San Jose, CA
• 24: R. Prelinger. www.prelinger.com, 2008. http://www.panix.com/~footage/.
• 25: T. Schwarz, M. Baker, S. Bassi, B. Baumgart, W. Flagg, C. van Ingen, K. Joste, M. Manasse, and M. Shah. Disk failure investigations at the internet archive. In MSST2006: 23rd IEEE, 14th NASA Goddard Conference on Mass Storage Systems and Technologies, May 2006.
• 26: S. Technology. Seagate technology - cheetah hard drive family, 2009. http://www.seagate.com/www/en-us/products/servers/cheetah/.
• 27: Yahoo!, 2008. http://developer.yahoo.net/blogs/hadoop/2008/02/yahoo-worlds-largest-production-hadoop.html.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures; Complexity measures