Cluster computing for web-scale data processing

ABSTRACT

In this paper we present the design of a modern course in cluster computing and large-scale data processing. The defining differences between this and previously published designs are its focus on processing very large data sets and its use of Hadoop, an open source Java-based implementation of MapReduce and the Google File System as the platform for programming exercises. Hadoop proved to be a key element for successfully implementing structured lab activities and independent design projects. Through this course, offered at the University of Washington in 2007, we imparted new skills on our students, improving their ability to design systems capable of solving web-scale problems.

REFERENCES

• 1: ACM/IEEE-CS Joint Curriculum Task Force. Computing Curricula 2001. IEEE Computer Society and Association for Computing Machinery., 2001.
• 2: David P. Anderson , Carl Christensen , Bruce Allen, Designing a runtime system for volunteer computing, Proceedings of the 2006 ACM/IEEE conference on Supercomputing, November 11-17, 2006, Tampa, Florida [doi>10.1145/1188455.1188586]
• 3: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 4: Mike Cafarella , Doug Cutting, Building Nutch: Open Source Search, Queue, v.2 n.2, April 2004 [doi>10.1145/988392.988408]
• 5: José C. Cunha , João Lourenço, An integrated course on parallel and distributed processing, Proceedings of the twenty-ninth SIGCSE technical symposium on Computer science education, p.217-221, February 26-March 01, 1998, Atlanta, Georgia, United States
• 6: Jeffrey Dean , Sanjay Ghemawat, MapReduce: simplified data processing on large clusters, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.10-10, December 06-08, 2004, San Francisco, CA
• 7: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, ACM SIGOPS Operating Systems Review, v.37 n.5, December 2003
• 8: Hadoop. http://lucene.apache.org/hadoop/.
• 9: Kimball and S. Michels-Slettvet. CSE 490H lecture notes: Problem solving on large scale clusters.
• 10: http://code.google.com/edu/content/submissions/
• 11: uwspr2007_clustercourse/listing.html, 2007.
• 12: Chuck Pheatt, An easy to use distributed computing framework, Proceedings of the 38th SIGCSE technical symposium on Computer science education, March 07-11, 2007, Covington, Kentucky, USA
• 13: Sahami. Scaling computer science education to education on scaling in computer science. Workshop on Integrative Computing Education & Research (ICER): Preparing IT Graduates for 2010 and Beyond, Jan. 2006.
• 14: M. Satyanarayanan , John H. Howard , David A. Nichols , Robert N. Sidebotham , Alfred Z. Spector , Michael J. West, The ITC distributed file system: principles and design, Proceedings of the tenth ACM symposium on Operating systems principles, p.35-50, December 1985, Orcas Island, Washington, United States

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education

General Terms:
Design, Experimentation

Keywords:
clusters, distributed computing, education, hadoop, mapreduce