Dryad

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Dryad: distributed data-parallel programs from sequential building blocks

Full text

Pdf (498 KB)

Source

European Conference on Computer Systems archive
Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 table of contents

Lisbon, Portugal

SESSION: Multi-core/multi-processor issues table of contents

Pages: 59 - 72

Year of Publication: 2007

ISBN ~ ISSN:0163-5980 , 978-1-59593-636-3

Also published in ...

Authors

Michael Isard	Microsoft Research, Silicon Valley
Mihai Budiu	Microsoft Research, Silicon Valley
Yuan Yu	Microsoft Research, Silicon Valley
Andrew Birrell	Microsoft Research, Silicon Valley
Dennis Fetterly	Microsoft Research, Silicon Valley

Sponsor

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 72, Downloads (12 Months): 417, Citation Count: 33

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1272996.1273005 What is a DOI?

ABSTRACT

Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad application combines computational "vertices" with communication "channels" to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through flies, TCP pipes, and shared-memory FIFOs.

The vertices provided by the application developer are quite simple and are usually written as sequential programs with no thread creation or locking. Concurrency arises from Dryad scheduling vertices to run simultaneously on multiple computers, or on multiple CPU cores within a computer. The application can discover the size and placement of data at run time, and modify the graph as the computation progresses to make efficient use of the available resources.

Dryad is designed to scale from powerful multi-core single computers, through small clusters of computers, to data centers with thousands of computers. The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Global grid forum. http://www.gridforum.org/.
	2	Intel IXP2XXX product line of network processors. http://www.intel.com/design/network/products/npfamily/ixp2xxx.htm.
	3	Intel platform 2015. http://www.Intel.com/technology/architecture/platform2015/.
	4	The LINQ project. http://msdn.microsoft.com/netframework/future/linq/.
	5	Open MPI. http://www.open-mpi.org/.
	6	SQL Server Integration Services. http://www.microsoft.com/sq1/technologies/integration/default.mspx.
	7	Thomas E. Anderson , David E. Culler , David A. Patterson , and the NOW team, A Case for NOW (Networks of Workstations), IEEE Micro, v.15 n.1, p.54-64, February 1995 [doi>10.1109/40.342018]
	8	Remzi H. Arpaci-Dusseau, Run-time adaptation in river, ACM Transactions on Computer Systems (TOCS), v.21 n.1, p.36-86, February 2003 [doi>10.1145/592637.592639]
	9	Özalp Babaoğlu , Lorenzo Alvisi , Alessandro Amoroso , Renzo Davoli , Luigi Alberto Giachini, Paralex: an environment for parallel programming in distributed systems, Proceedings of the 6th international conference on Supercomputing, p.178-187, July 19-24, 1992, Washington, D. C., United States [doi>10.1145/143369.143406]
	10	Magdalena Balazinska , Hari Balakrishnan , Samuel Madden , Michael Stonebraker, Fault-tolerance in the Borealis distributed stream processing system, Proceedings of the 2005 ACM SIGMOD international conference on Management of data, June 14-16, 2005, Baltimore, Maryland [doi>10.1145/1066157.1066160]
	11	Tom Barclay , Robert Barnes , Jim Gray , Prakash Sundaresan, Loading databases using dataflow parallelism, ACM SIGMOD Record, v.23 n.4, p.72-83, Dec. 1994 [doi>10.1145/190627.190647]
	12	Chaitanya Baru , Gilles Fecteau, An overview of DB2 parallel edition, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.460-462, May 22-25, 1995, San Jose, California, United States
	13	Guy E. Blelloch, Programming parallel algorithms, Communications of the ACM, v.39 n.3, p.85-97, March 1996 [doi>10.1145/227234.227246]
	14	Robert D. Blumofe , Christopher F. Joerg , Bradley C. Kuszmaul , Charles E. Leiserson , Keith H. Randall , Yuli Zhou, Cilk: an efficient multithreaded runtime system, Proceedings of the fifth ACM SIGPLAN symposium on Principles and practice of parallel programming, p.207-216, July 19-21, 1995, Santa Barbara, California, United States
	15	Eylon Caspi , Michael Chu , Randy Huang , Joseph Yeh , John Wawrzynek , André DeHon, Stream Computations Organized for Reconfigurable Execution (SCORE), Proceedings of the The Roadmap to Reconfigurable Computing, 10th International Workshop on Field-Programmable Logic and Applications, p.605-614, August 27-30, 2000
	16	Jeffrey Dean , Sanjay Ghemawat, MapReduce: simplified data processing on large clusters, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.10-10, December 06-08, 2004, San Francisco, CA
	17	D. J. Dewitt , S. Ghandeharizadeh , D. A. Schneider , A. Bricker , H. -I. Hsiao , R. Rasmussen, The Gamma Database Machine Project, IEEE Transactions on Knowledge and Data Engineering, v.2 n.1, p.44-62, March 1990 [doi>10.1109/69.50905]
	18	David DeWitt , Jim Gray, Parallel database systems: the future of high performance database systems, Communications of the ACM, v.35 n.6, p.85-98, June 1992 [doi>10.1145/129888.129894]
	19	D. C. DiNucci and R. G. Babb II. Design and implementation of parallel programs with LGDF2. In Digest of Papers from Compcon '89, pages 102--107, 1989.
	20	Armando Fox , Steven D. Gribble , Yatin Chawathe , Eric A. Brewer , Paul Gauthier, Cluster-based scalable network services, Proceedings of the sixteenth ACM symposium on Operating systems principles, p.78-91, October 05-08, 1997, Saint Malo, France
	21	Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
	22	Goetz Graefe, Encapsulation of parallelism in the Volcano query processing system, Proceedings of the 1990 ACM SIGMOD international conference on Management of data, p.102-111, May 23-26, 1990, Atlantic City, New Jersey, United States
	23	J. Gray, A. S. Szalay, A. Thakar, P. Kunszt, C. Stoughton, D. Slutz, and J. Vandenberg. Data mining the SDSS SkyServer database. In Distributed Data and Structures 4: Records of the 4th International Meeting, pages 189--210, Paris, France, March 2002. Carleton Scientific, also as MSR-TR-2002-01.
	24	Jim Gray and Alex Szalay. Science in an exponential world. Nature, 440(23), March 23 2006.
	25	J.-H. Hwang, M. Balazinska, A. Rasin, U. Çetintemel, M. Stonebraker, and S. Zdonik. A comparison of stream-oriented high-availability algorithms. Technical Report TR-03-17, Computer Science Department, Brown University, September 2003.
	26	The Imagine Stream Processor, Proceedings of the 2002 IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD'02), p.282, September 16-18, 2002
	27	Eddie Kohler , Robert Morris , Benjie Chen , John Jannotti , M. Frans Kaashoek, The click modular router, ACM Transactions on Computer Systems (TOCS), v.18 n.3, p.263-297, Aug. 2000 [doi>10.1145/354871.354874]
	28	James R. Larus , Michael Parkes, Using Cohort-Scheduling to Enhance Server Performance, Proceedings of the General Track: 2002 USENIX Annual Technical Conference, p.103-114, June 10-15, 2002
	29	Orlando Loques , Julius Leite , Enrique Vinicio Carrera E., P-RIO: A Modular Parallel-Programming Environment, IEEE Concurrency, v.6 n.1, p.47-57, January 1998 [doi>10.1109/4434.656779]
	30	William R. Mark , R. Steven Glanville , Kurt Akeley , Mark J. Kilgard, Cg: a system for programming graphics hardware in a C-like language, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
	31	Peter Newton , James C. Browne, The CODE 2.0 graphical parallel programming language, Proceedings of the 6th international conference on Supercomputing, p.167-177, July 19-24, 1992, Washington, D. C., United States [doi>10.1145/143369.143405]
	32	Ken Phillips. SenSage ESA. SC Magazine, March 1 2006.
	33	Rob Pike , Sean Dorward , Robert Griesemer , Sean Quinlan, Interpreting the data: Parallel analysis with Sawzall, Scientific Programming, v.13 n.4, p.277-298, October 2005
	34	Mehul A. Shah , Joseph M. Hellerstein , Eric Brewer, Highly available, fault-tolerant, parallel dataflows, Proceedings of the 2004 ACM SIGMOD international conference on Management of data, June 13-18, 2004, Paris, France [doi>10.1145/1007568.1007662]
	35	V. S. Sunderam, PVM: a framework for parallel distributed computing, Concurrency: Practice and Experience, v.2 n.4, p.315-339, Dec. 1990 [doi>10.1002/cpe.4330020404]
	36	David Tarditi , Sidd Puri , Jose Oglesby, Accelerator: using data parallelism to program GPUs for general-purpose uses, Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, October 21-25, 2006, San Jose, California, USA
	37	Douglas Thain , Todd Tannenbaum , Miron Livny, Distributed computing in practice: the Condor experience: Research Articles, Concurrency and Computation: Practice & Experience, v.17 n.2-4, p.323-356, February 2005 [doi>10.1002/cpe.v17:2/4]
	38	P. W. Trinder , H.-W. Loidl , R. F. Pointon, Parallel and Distributed Haskells, Journal of Functional Programming, v.12 n.5, p.469-510, July 2002 [doi>10.1017/S0956796802004343]

CITED BY 33

	Christopher Olston , Benjamin Reed , Utkarsh Srivastava , Ravi Kumar , Andrew Tomkins, Pig latin: a not-so-foreign language for data processing, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada
	Brandon Rich , Douglas Thain, DataLab: transactional data-parallel computing on an active storage cloud, Proceedings of the 17th international symposium on High performance distributed computing, June 23-27, 2008, Boston, MA, USA
	Daniel Marino , Madanlal Musuvathi , Satish Narayanasamy, LiteRace: effective sampling for lightweight data-race detection, ACM SIGPLAN Notices, v.44 n.6, June 2009
	Autonomic operations in cooperative stream processing systems, Hot Topics in Autonomic Computing on Hot Topics in Autonomic Computing, p.9-9, June 15, 2007, Jacksonville, FL
	Richard M. C. McCreadie , Craig Macdonald , Iadh Ounis, On single-pass indexing with MapReduce, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
	Madanlal Musuvathi , Shaz Qadeer, Fair stateless model checking, ACM SIGPLAN Notices, v.43 n.6, June 2008
	Wei Lu , Thilina Gunarathne , Dennis Gannon, Developing a concurrent service orchestration engine in ccr, Proceedings of the 1st international workshop on Multicore software engineering, May 11-11, 2008, Leipzig, Germany
	Felix Weigel , Biswanath Panda , Mirek Riedewald , Johannes Gehrke , Manuel Calimlim, Large-scale collaborative analysis and extraction of web data, Proceedings of the VLDB Endowment, v.1 n.2, August 2008
	Xuezheng Liu , Zhenyu Guo , Xi Wang , Feibo Chen , Xiaochen Lian , Jian Tang , Ming Wu , M. Frans Kaashoek , Zheng Zhang, D³S: debugging deployed distributed systems, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.423-437, April 16-18, 2008, San Francisco, California
	Jayanth Gummaraju , Joel Coburn , Yoshio Turner , Mendel Rosenblum, Streamware: programming general-purpose multicore processors using streams, ACM SIGOPS Operating Systems Review, v.42 n.2, March 2008
	Dionysios Logothetis , Kenneth Yocum, Ad-hoc data processing in the cloud, Proceedings of the VLDB Endowment, v.1 n.2, August 2008
	Ronnie Chaiken , Bob Jenkins , Per-Åke Larson , Bill Ramsey , Darren Shakib , Simon Weaver , Jingren Zhou, SCOPE: easy and efficient parallel processing of massive data sets, Proceedings of the VLDB Endowment, v.1 n.2, August 2008
	Parag Agrawal , Daniel Kifer , Christopher Olston, Scheduling shared scans of large data files, Proceedings of the VLDB Endowment, v.1 n.1, August 2008
	David J. DeWitt , Erik Paulson , Eric Robinson , Jeffrey Naughton , Joshua Royalty , Srinath Shankar , Andrew Krioukov, Clustera: an integrated computation and data management system, Proceedings of the VLDB Endowment, v.1 n.1, August 2008
	Chuanxiong Guo , Haitao Wu , Kun Tan , Lei Shi , Yongguang Zhang , Songwu Lu, Dcell: a scalable and fault-tolerant network structure for data centers, ACM SIGCOMM Computer Communication Review, v.38 n.4, October 2008
	Ioan Raicu , Zhao Zhang , Mike Wilde , Ian Foster , Pete Beckman , Kamil Iskra , Ben Clifford, Toward loosely coupled programming on petascale systems, Proceedings of the 2008 ACM/IEEE conference on Supercomputing, November 15-21, 2008, Austin, Texas
	Alexander Rasmussen , Emre Kiciman , Benjamin Livshits , Madanlal Musuvathi, Improving the responsiveness of internet services with automatic cache placement, Proceedings of the fourth ACM european conference on Computer systems, April 01-03, 2009, Nuremberg, Germany
	Adrian M. Caulfield , Laura M. Grupp , Steven Swanson, Gordon: using flash memory to build fast, power-efficient clusters for data-intensive applications, ACM SIGPLAN Notices, v.44 n.3, March 2009
	Michael A. Kozuch , Michael P. Ryan , Richard Gass , Steven W. Schlosser , David O'Hallaron , James Cipar , Elie Krevat , Julio López , Michael Stroucken , Gregory R. Ganger, Tashi: location-aware cluster management, Proceedings of the 1st workshop on Automated control for datacenters and clouds, June 19-19, 2009, Barcelona, Spain
	Li Yi , Christopher Moretti , Scott Emrich , Kenneth Judd , Douglas Thain, Harnessing parallelism in multicore clusters with the all-pairs and wavefront abstractions, Proceedings of the 18th ACM international symposium on High performance distributed computing, June 11-13, 2009, Garching, Germany
	Yao Zhao , Yinglian Xie , Fang Yu , Qifa Ke , Yuan Yu , Yan Chen , Eliot Gillum, BotGraph: large scale spamming botnet detection, Proceedings of the 6th USENIX symposium on Networked systems design and implementation, p.321-334, April 22-24, 2009, Boston, Massachusetts
	Jan S. Rellermeyer , Michael Duller , Gustavo Alonso, Engineering the cloud from software modules, Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing, p.32-37, May 23-23, 2009
	Thomas Sandholm , Kevin Lai, MapReduce optimization using regulated dynamic prioritization, Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems, June 15-19, 2009, Seattle, WA, USA
	Mumtaz Ahmad , Ashraf Aboulnaga , Shivnath Babu, Query interactions in database workloads, Proceedings of the Second International Workshop on Testing Database Systems, June 29-29, 2009, Providence, Rhode Island
	Padmanabhan S. Pillai , Lily B. Mummert , Steven W. Schlosser , Rahul Sukthankar , Casey J. Helfrich, SLIPstream: scalable low-latency interactive perception on streaming data, Proceedings of the 18th international workshop on Network and operating systems support for digital audio and video, June 03-05, 2009, Williamsburg, VA, USA
	Michael Isard , Yuan Yu, Distributed data-parallel computing using a high-level programming language, Proceedings of the 35th SIGMOD international conference on Management of data, June 29-July 02, 2009, Providence, Rhode Island, USA
	Andrew Pavlo , Erik Paulson , Alexander Rasin , Daniel J. Abadi , David J. DeWitt , Samuel Madden , Michael Stonebraker, A comparison of approaches to large-scale data analysis, Proceedings of the 35th SIGMOD international conference on Management of data, June 29-July 02, 2009, Providence, Rhode Island, USA
	Christopher Olston , Shubham Chopra , Utkarsh Srivastava, Generating example data for dataflow programs, Proceedings of the 35th SIGMOD international conference on Management of data, June 29-July 02, 2009, Providence, Rhode Island, USA
	Edward Bortnikov, Open-source grid technologies for web-scale computing, ACM SIGACT News, v.40 n.2, June 2009
	Joe B. Buck , Noah Watkins , Carlos Maltzahn , Scott A. Brandt, Abstract storage: moving file format-specific abstractions intopetabyte-scale storage systems, Proceedings of the second international workshop on Data-aware distributed computing, p.31-40, June 09-10, 2009, Garching, Germany
	Chuanxiong Guo , Guohan Lu , Dan Li , Haitao Wu , Xuan Zhang , Yunfeng Shi , Chen Tian , Yongguang Zhang , Songwu Lu, BCube: a high performance, server-centric network architecture for modular data centers, ACM SIGCOMM Computer Communication Review, v.39 n.4, October 2009
	Noah Paul Evans , Eric Van Hensbergen, Brief announcement: PUSH, a DISC shell, Proceedings of the 28th ACM symposium on Principles of distributed computing, August 10-12, 2009, Calgary, AB, Canada
	Frank D. McSherry, Privacy integrated queries: an extensible platform for privacy-preserving data analysis, Proceedings of the 35th SIGMOD international conference on Management of data, June 29-July 02, 2009, Providence, Rhode Island, USA