Provenance and scientific workflows: challenges and opportunities

ABSTRACT

Provenance in the context of workflows, both for the data they derive and for their specification, is an essential component to allow for result reproducibility, sharing, and knowledge re-use in the scientific community. Several workshops have been held on the topic, and it has been the focus of many research projects and prototype systems. This tutorial provides an overview of research issues in provenance for scientific workflows, with a focus on recent literature and technology in this area. It is aimed at a general database research audience and at people who work with scientific data and workflows. We will (1) provide a general overview of scientific workflows, (2) describe research on provenance for scientific workflows and show in detail how provenance is supported in existing systems; (3) discuss emerging applications that are enabled by provenance; and (4) outline open problems and new directions for database-related research.

REFERENCES

• 1: Workflow management: models, methods, and systems, MIT Press, Cambridge, MA, 2002
• 2: I. Altintas, O. Barney, and E. Jaeger-Frank. Provenance collection support in the kepler scientific workflow system. In Proceedings of the International Provenance and Annotation Workshop (IPAW), pages 118--132, 2006.
• 3: Roger S. Barga , Luciano A. Digiampietri, Automatic capture and efficient storage of e-Science experiment provenance, Concurrency and Computation: Practice & Experience, v.20 n.5, p.419-429, April 2008 [doi>10.1002/cpe.v20:5]
• 4: Catriel Beeri , Anat Eyal , Simon Kamenkovich , Tova Milo, Querying business processes, Proceedings of the 32nd international conference on Very large data bases, September 12-15, 2006, Seoul, Korea
• 5: O. Biton, S. Cohen-Boulakia, S. Davidson, and C. Hara. Querying and managing provenance through user views in scientific workflows. In Proceedings of ICDE, 2008. To appear.
• 6: Rajendra Bose , Ian Foster , Luc Moreau, Report on the International Provenance and Annotation Workshop: (IPAW'06) 3-5 May 2006, Chicago, ACM SIGMOD Record, v.35 n.3, September 2006 [doi>10.1145/1168092.1168102]
• 7: Rajendra Bose , James Frew, Lineage retrieval for scientific data processing: a survey, ACM Computing Surveys (CSUR), v.37 n.1, p.1-28, March 2005 [doi>10.1145/1057977.1057978]
• 8: Shawn Bowers , Timothy M. McPhillips , Bertram Ludäscher, Provenance in collection-oriented scientific workflows, Concurrency and Computation: Practice & Experience, v.20 n.5, p.519-529, April 2008 [doi>10.1002/cpe.v20:5]
• 9: Business Process Execution Language for Web Services. http://www.ibm.com/developerworks/library/specification/ws-bpel/.
• 10: Peter Buneman , Wang-Chiew Tan, Provenance in databases, Proceedings of the 2007 ACM SIGMOD international conference on Management of data, June 11-14, 2007, Beijing, China [doi>10.1145/1247480.1247646]
• 11: Ben Clifford , Ian Foster , Jens-S. Voeckler , Michael Wilde , Yong Zhao, Tracking provenance in a virtual data grid, Concurrency and Computation: Practice & Experience, v.20 n.5, p.565-575, April 2008 [doi>10.1002/cpe.v20:5]
• 12: S. Cohen, S. C. Boulakia, and S. B. Davidson. Towards a model of provenance and user views in scientific workflows. In DILS, pages 264--279, 2006.
• 13: Sarah Cohen-Boulakia , Olivier Biton , Shirley Cohen , Susan Davidson, Addressing the provenance challenge using ZOOM, Concurrency and Computation: Practice & Experience, v.20 n.5, p.497-506, April 2008 [doi>10.1002/cpe.v20:5]
• 14: S. B. Davidson, S. C. Boulakia, A. Eyal, B. Ludäscher, T. M. McPhillips, S. Bowers, M. K. Anand, and J. Freire. Provenance in scientific workflow systems. IEEE Data Eng. Bull., 30(4):44--50, 2007.
• 15: E. Deelman and Y. Gil. NSF Workshop on Challenges of Scientific Workflows. Technical report, NSF, 2006. http://vtcpc.isi.edu/wiki/index.php/Main_Page.
• 16: Ewa Deelman , Gurmeet Singh , Mei-Hui Su , James Blythe , Yolanda Gil , Carl Kesselman , Gaurang Mehta , Karan Vahi , G. Bruce Berriman , John Good , Anastasia Laity , Joseph C. Jacob , Daniel S. Katz, Pegasus: A framework for mapping complex scientific workflows onto distributed systems, Scientific Programming, v.13 n.3, p.219-237, July 2005
• 17: Ian T. Foster , Jens-S. Vöckler , Michael Wilde , Yong Zhao, Chimera: AVirtual Data System for Representing, Querying, and Automating Data Derivation, Proceedings of the 14th International Conference on Scientific and Statistical Database Management, p.37-46, July 24-26, 2002 [doi>10.1109/SSDM.2002.1029704]
• 18: Juliana Freire , David Koop , Emanuele Santos , Cláudio T. Silva, Provenance for Computational Tasks: A Survey, Computing in Science and Engineering, v.10 n.3, p.11-21, May 2008 [doi>10.1109/MCSE.2008.79]
• 19: Fernanda B. Viégas , Martin Wattenberg , Jeffrey Heer , Maneesh Agrawala, Social data analysis workshop, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358971]
• 20: J. Freire, C. T. Silva, S. P. Callahan, E. Santos, C. E. Scheidegger, and H. T. Vo. Managing rapidly-evolving scientific workflows. In International Provenance and Annotation Workshop (IPAW), LNCS 4145, pages 10--18, 2006. Invited paper.
• 21: D. Gannon et al. A Workshop on Scientific and Scholarly Workflow Cyberinfrastructure: Improving Interoperability, Sustainability and Platform Convergence in Scientific And Scholarly Workflow. Technical report, NSF and Mellon Foundation, 2007. https://spaces.internet2.edu/display/SciSchWorkflow.
• 22: Jennifer Golbeck , James Hendler, A Semantic Web approach to the provenance challenge, Concurrency and Computation: Practice & Experience, v.20 n.5, p.431-439, April 2008 [doi>10.1002/cpe.v20:5]
• 23: L. Haas. Information for people. http://www.almaden.ibm.com/cs/people/laura/ Information For People keynote.pdf, 2007. Keynote talk at ICDE.
• 24: H. V. Jagadish. Making database systems usable. http://www.eecs.umich.edu/db/usable/ usability-sigmod.ppt, 2007. Keynote talk at SIGMOD.
• 25: The Kepler Project. http://kepler-project.org.
• 26: Jihie Kim , Ewa Deelman , Yolanda Gil , Gaurang Mehta , Varun Ratnakar, Provenance trails in the Wings-Pegasus system, Concurrency and Computation: Practice & Experience, v.20 n.5, p.587-597, April 2008 [doi>10.1002/cpe.v20:5]
• 27: Microsoft Workflow Foundation. http://msdn2.microsoft.com/en-us/netframework/ aa663322.aspx.
• 28: Simon Miles , Paul Groth , Steve Munroe , Sheng Jiang , Thibaut Assandri , Luc Moreau, Extracting causal graphs from an open provenance data model, Concurrency and Computation: Practice & Experience, v.20 n.5, p.577-586, April 2008 [doi>10.1002/cpe.v20:5]
• 29: Luc Moreau , Ian Foster, Provenance and Annotation of Data: International Provenance and Annotation Workshop, IPAW 2006, Chicago, Il, USA, May 3-5, 2006, Revised Selected Papers (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 30: L. Moreau, J. Freire, J. Futrelle, R. McGrath, J. Myers, and P. Paulson. The open provenance model, December 2007. http://eprints.ecs.soton.ac.uk/14979.
• 31: Steven G. Parker , Christopher R. Johnson, SCIRun: a scientific programming environment for computational steering, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.52-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224354]
• 32: First provenance challenge. http://twiki.ipaw.info/bin/view/Challenge/ FirstProvenanceChallenge, 2006. S. Miles, and L. Moreau (organizers).
• 33: Second provenance challenge. http://twiki.ipaw.info/bin/view/Challenge/ SecondProvenanceChallenge, 2007. J. Freire, S. Miles, and L. Moreau (organizers).
• 34: Carlos Scheidegger , Huy Vo , David Koop , Juliana Freire , Claudio Silva, Querying and Creating Visualizations by Analogy, IEEE Transactions on Visualization and Computer Graphics, v.13 n.6, p.1560-1567, November 2007 [doi>10.1109/TVCG.2007.70584]
• 35: Claudio T. Silva , Juliana Freire , Steven P. Callahan, Provenance for Visualizations: Reproducibility and Beyond, Computing in Science and Engineering, v.9 n.5, p.82-89, September 2007 [doi>10.1109/MCSE.2007.106]
• 36: Yogesh L. Simmhan , Beth Plale , Dennis Gannon, A survey of data provenance in e-science, ACM SIGMOD Record, v.34 n.3, September 2005 [doi>10.1145/1084805.1084812]
• 37: Y. L. Simmhan, B. Plale, and D. Gannon. Karma2: Provenance management for data driven workflows. International Journal of Web Services Research, Idea Group Publishing, 5:1, 2008. To Appear.
• 38: Y. L. Simmhan, B. Plale, D. Gannon, and S. Marru. Performance evaluation of the karma provenance framework for scientific workflows. In L. Moreau and I. T. Foster, editors, International Provenance and Annotation Workshop (IPAW), Chicago, IL, volume 4145 of Lecture Notes in Computer Science, pages 222--236. Springer, 2006.
• 39: The Swift System. www.ci.uchicago.edu/swift.
• 40: W. C. Tan. Provenance in databases: Past, current, and future. IEEE Data Eng. Bull., 30(4):3--12, 2007.
• 41: The Taverna Project. http://taverna.sourceforge.net.
• 42: The Triana Project. http://www.trianacode.org.
• 43: VDS - The GriPhyN Virtual Data System. http://www.ci.uchicago.edu/wiki/bin/view/VDS/VDSWeb/WebMain.
• 44: Fernanda B. Viegas , Martin Wattenberg , Frank van Ham , Jesse Kriss , Matt McKeon, ManyEyes: a Site for Visualization at Internet Scale, IEEE Transactions on Visualization and Computer Graphics, v.13 n.6, p.1121-1128, November 2007 [doi>10.1109/TVCG.2007.70577]
• 45: The VisTrails Project. http://www.vistrails.org.
• 46: Jun Zhao , Carole Goble , Robert Stevens , Daniele Turi, Mining Taverna's semantic web of provenance, Concurrency and Computation: Practice & Experience, v.20 n.5, p.463-472, April 2008 [doi>10.1002/cpe.v20:5]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.0 General

General Terms:
Documentation, Experimentation

Keywords:
provenance, scientific workflows