Improving file system reliability with I/O shepherding

ABSTRACT

We introduce a new reliability infrastructure for file systems called I/O shepherding. I/O shepherding allows a file system developer to craft nuanced reliability policies to detect and recover from a wide range of storage system failures. We incorporate shepherding into the Linux ext3 file system through a set of changes to the consistency management subsystem, layout engine, disk scheduler, and buffer cache. The resulting file system, CrookFS, enables a broad class of policies to be easily and correctly specified. We implement numerous policies, incorporating data protection techniques such as retry, parity, mirrors, checksums, sanity checks, and data structure repairs; even complex policies can be implemented in less than 100 lines of code, confirming the power and simplicity of the shepherding framework. We also demonstrate that shepherding is properly integrated, adding less than 5% overhead to the I/O path.

REFERENCES

• 1: David Andersen , Deepak Bansal , Dorothy Curtis , Srinivasan Seshan , Hari Balakrishnan, System support for bandwidth management and content adaptation in internet applications, Proceedings of the 4th conference on Symposium on Operating System Design & Implementation, p.15-15, October 22-25, 2000, San Diego, California
• 2: Dave Anderson , Jim Dykes , Erik Riedel, More Than an Interface---SCSI vs. ATA, Proceedings of the 2nd USENIX Conference on File and Storage Technologies, March 31-31, 2003, San Francisco, CA
• 3: Lakshmi Bairavasundaram. On the frequency of transient faults in modern disk drives. Personal Communication, 2007.
• 4: Lakshmi N. Bairavasundaram , Garth R. Goodson , Shankar Pasupathy , Jiri Schindler, An analysis of latent sector errors in disk drives, Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 12-16, 2007, San Diego, California, USA
• 5: Hari Balakrishnan , Hariharan S. Rahul , Srinivasan Seshan, An integrated congestion management architecture for Internet hosts, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.175-187, August 30-September 03, 1999, Cambridge, Massachusetts, United States
• 6: Wendy Bartlett , Lisa Spainhower, Commercial Fault Tolerance: A Tale of Two Systems, IEEE Transactions on Dependable and Secure Computing, v.1 n.1, p.87-96, January 2004 [doi>10.1109/TDSC.2004.4]
• 7: Aaron B. Brown , David A. Patterson, Undo for operators: building an undoable e-mail store, Proceedings of the annual conference on USENIX Annual Technical Conference, p.1-1, June 09-14, 2003, San Antonio, Texas
• 8: Andy Chou , Junfeng Yang , Benjamin Chelf , Seth Hallem , Dawson Engler, An empirical study of operating systems errors, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 9: Yvonne Coady , Gregor Kiczales , Mike Feeley , Greg Smolyn, Using aspectC to improve the modularity of path-specific customization in operating system code, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, September 10-14, 2001, Vienna, Austria
• 10: Peter Corbett , Bob English , Atul Goel , Tomislav Grcanac , Steven Kleiman , James Leong , Sunitha Sankar, Awarded Best Paper! -- Row-Diagonal Parity for Double Disk Failure Correction, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 11: Timothy E. Denehy , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, Bridging the Information Gap in Storage Protocol Stacks, Proceedings of the General Track of the annual conference on USENIX Annual Technical Conference, p.177-190, June 10-15, 2002
• 12: Dawson Engler , David Yu Chen , Seth Hallem , Andy Chou , Benjamin Chelf, Bugs as deviant behavior: a general approach to inferring errors in systems code, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 13: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 14: Jim Gray. A Census of Tandem System Availability Between 1985 and 1990. Technical Report 90.1, Tandem Computers, 1990.
• 15: Jim Gray , Andreas Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1992
• 16: Roedy Green. EIDE Controller Flaws Version 24. http://mindprod.com/jgloss/eideflaw.html, February 2005.
• 17: R. Hagmann, Reimplementing the Cedar file system using logging and group commit, Proceedings of the eleventh ACM Symposium on Operating systems principles, p.155-162, November 08-11, 1987, Austin, Texas, United States
• 18: John S. Heidemann , Gerald J. Popek, File-system development with stackable layers, ACM Transactions on Computer Systems (TOCS), v.12 n.1, p.58-89, Feb. 1994 [doi>10.1145/174613.174616]
• 19: Hai Huang , Wanda Hung , Kang G. Shin, FS2: dynamic data replication in free disk space for improving disk performance and energy consumption, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 20: Gordon F. Hughes , Joseph F. Murray, Reliability and security of RAID storage systems and D2D archives using SATA disk drives, ACM Transactions on Storage (TOS), v.1 n.1, p.95-107, February 2005 [doi>10.1145/1044956.1044961]
• 21: Hannu Kari , Heikki Saikkonen , Fabrizio Lombardi, Detection of Defective Media in Disks, Proceedings of the IEEE International Workshop on Defect and Fault Tolerance in VLSI Systems, p.49-55, October 27-29, 1993
• 22: Jeffrey Katcher. PostMark: A New File System Benchmark. Technical Report TR-3022, Network Appliance Inc., October 1997.
• 23: Gregor Kiczales, John Lamping, Anurag Mendhekar, Chris Maeda, Cristina Lopes, Jean-Marc Loingtier, and John Irwin. Aspect-Oriented Programming. In Proceedings of the European Conference on Object-Oriented Programming (ECOOP), pages 220--242, 1997.
• 24: Steve R. Kleiman. Vnodes: An Architecture for Multiple File System Types in Sun UNIX. In USENIX Summer'86, pages 238--247, Atlanta, GA, June 1986.
• 25: Henry F. Korth , Eliezer levy , Abraham Silberschatz, A formal approach to recovery by compensating transactions, Proceedings of the sixteenth international conference on Very large databases, p.95-106, September 1990, Brisbane, Australia
• 26: Larry Lancaster , Alan Rowe, Measuring Real-World Data Availability, Proceedings of the 15th USENIX conference on System administration, December 02-07, 2001, San Diego, California
• 27: Robert Morris , Eddie Kohler , John Jannotti , M. Frans Kaashoek, The Click modular router, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.217-231, December 12-15, 1999, Charleston, South Carolina, United States
• 28: Kiran Nagaraja , Fábio Oliveira , Ricardo Bianchini , Richard P. Martin , Thu D. Nguyen, Understanding and dealing with operator mistakes in internet services, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.5-5, December 06-08, 2004, San Francisco, CA
• 29: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 30: Vijayan Prabhakaran , Lakshmi N. Bairavasundaram , Nitin Agrawal , Haryadi S. Gunawi , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, IRON file systems, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 31: Bianca Schroeder , Garth A. Gibson, Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?, Proceedings of the 5th USENIX conference on File and Storage Technologies, p.1-es, February 13-16, 2007, San Jose, CA
• 32: Muthian Sivathanu , Vijayan Prabhakaran , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau, Awarded Best Student Paper! -- Improving Storage System Availability with D-GRAID, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 33: Sun Microsystems. ZFS: The last word in file systems. www.sun.com/2004-0914/feature/, 2006.
• 34: Rajesh Sundaram. The Private Lives of Disk Drives. http://www.netapp.com/go/techontap/matl/sample/0206tot_resiliency.html, February 2006.
• 35: Michael M. Swift , Brian N. Bershad , Henry M. Levy, Improving the reliability of commodity operating systems, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 36: Nisha Talagala and David Patterson. An Analysis of Error Behaviour in a Large Storage System. In The IEEE Workshop on Fault Tolerance in Parallel and Distributed Systems, San Juan, Puerto Rico, April 1999.
• 37: Transaction Processing Council. TPC Benchmark B Standard Specification, Revision 3.2. Technical Report, 1990.
• 38: Stephen C. Tweedie. Journaling the Linux ext2fs File System. In The Fourth Annual Linux Expo, Durham, North Carolina, May 1998.
• 39: Xiang Yu , Benjamin Gum , Yuqun Chen , Randolph Y. Wang , Kai Li , Arvind Krishnamurthy , Thomas E. Anderson, Trading capacity for performance in a disk array, Proceedings of the 4th conference on Symposium on Operating System Design & Implementation, p.17-17, October 22-25, 2000, San Diego, California
• 40: Erez Zadok , Jason Nieh, FiST: a language for stackable file systems, Proceedings of the annual conference on USENIX Annual Technical Conference, p.5-5, June 18-23, 2000, San Diego, California

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability

General Terms:
Design, Experimentation, Reliability

Keywords:
I/O shepherding, fault tolerance, reliability, storage