Versatility and Unix semantics in namespace unification

ABSTRACT

Administrators often prefer to keep related sets of files in different locations or media, as it is easier to maintain them separately. Users, however, prefer to see all files in one location for convenience. One solution that accommodates both needs is virtual namespace unification---providing a merged view of several directories without physically merging them. For example, namespace unification can merge the contents of several CD-ROM images without unpacking them, merge binary directories from different packages, merge views from several file servers, and more. Namespace unification can also enable snapshotting by marking some data sources read-only and then utilizing copy-on-write for the read-only sources. For example, an OS image may be contained on a read-only CD-ROM image---and the user's configuration, data, and programs could be stored in a separate read-write directory. With copy-on-write unification, the user need not be concerned about the two disparate file systems.It is difficult to maintain Unix semantics while offering a versatile namespace unification system. Past efforts to provide such unification often compromised on the set of features provided or Unix compatibility---resulting in an incomplete solution that users could not use.We designed and implemented a versatile namespace unification system called Unionfs. Unionfs maintains Unix semantics while offering advanced namespace unification features: dynamic insertion and removal of namespaces at any point in the merged view, mixing read-only and read-write components, efficient in-kernel duplicate elimination, NFS interoperability, and more. Since releasing our Linux implementation, it has been used by thousands of users and over a dozen Linux distributions, which helped us discover and solve many practical problems.

REFERENCES

• 1: AT&T Bell Laboratories. 1995. Plan 9---Programmer's Manual. AT&T Bell Laboratories.
• 2: Ellard, D. and Seltzer, M. 2003. NFS tricks and benchmarking traps. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track (San Antonio, Tex.), 101--114.
• 3: Flouris, M. D. and Bilas, A. 2004. Clotho: Transparent data versioning at the block I/O level. In Proceedings of the 12th NASA Goddard, 21st IEEE Conference on Mass Storage Systems and Technologies (MSST 2004). College Park, Md. 315--328.
• 4: Fraser, T., Badger, L., and Feldman, M. 1999. Hardening COTS software with generic software wrappers. In Proceedings of the 1999 IEEE Symposium on Security and Privacy, 2--16.
• 5: Guy, R. G., Heidemann, J. S., Mak, W., Page Jr., T. W., Popek, G. J., and Rothmeier, D. 1990. Implementation of the Ficus replicated file system. In Proceedings of the Summer USENIX Technical Conference, 63--71.
• 6: John S. Heidemann , Gerald J. Popek, File-system development with stackable layers, ACM Transactions on Computer Systems (TOCS), v.12 n.1, p.58-89, Feb. 1994 [doi>10.1145/174613.174616]
• 7: Hendricks, D. 1990. A filesystem for software development. In Proceedings of the USENIX Summer Conference (Anaheim, Calif.), 333--340.
• 8: Hitz, D., Lau, J., and Malcolm, M. 1994. File system design for an NFS file server appliance. In Proceedings of the USENIX Winter Technical Conference (San Francisco, Calif.), 235--245.
• 9: Joukov, N., Wright, C. P., and Zadok, E. 2004. FSprof: An in-kernel file system operations profiler. Tech. Rep. FSL-04-06, Computer Science Department, Stony Brook University. www.fsl.cs.sunysb.edu/docs/aggregate_stats-tr/aggregate_stats.pdf.
• 10: Kamp, P. H. and Watson, R. N. M. 2000. Jails: Confining the omnipotent root. In Proceedings of the 2nd International System Administration and Networking Conference (SANE2000). Maastricht, The Netherlands.
• 11: Katcher, J. 1997. PostMark: A new filesystem benchmark. Tech. Rep. TR3022, Network Appliance. www.netapp.com/tech_library/3022.html.
• 12: D. G. Korn , E. Krell, A new dimension for the UNIX file system, Software—Practice & Experience, v.20 n.S1, p.19-34, June 1990 [doi>10.1002/spe.4380201304]
• 13: McKusick, M. K. and Ganger, G. R. 1999. Soft updates: A technique for eliminating most synchronous writes in the fast filesystem. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track (Monterey, Calif.), 1--18.
• 14: OpenBSD. 2005. OpenSSH. www.openssh.org.
• 15: Pendry, J. S. and McKusick, M. K. 1995. Union mounts in 4.4BSD-Lite. In Proceedings of the USENIX Technical Conference on UNIX and Advanced Computing Systems, 25--33.
• 16: Pendry, J. S., Williams, N., and Zadok, E. 2003. Am-utils User Manual, 6.1b3 ed. www.am-utils.org.
• 17: Zachary Peterson , Randal Burns, Ext3cow: a time-shifting file system for regulatory compliance, ACM Transactions on Storage (TOS), v.1 n.2, p.190-212, May 2005 [doi>10.1145/1063786.1063789]
• 18: Roselli, D., Lorch, J. R., and Anderson, T. E. 2000. A comparison of file system workloads. In Proceedings of the Annual USENIX Technical Conference (San Diego, Calif.), 41--54.
• 19: Rosenthal, D. S. H. 1990. Evolving the Vnode interface. In Proceedings of the Summer USENIX Technical Conference, 107--18.
• 20: Joseph Spadavecchia , Erez Zadok, Enhancing NFS Cross-Administrative Domain Access, Proceedings of the FREENIX Track: 2002 USENIX Annual Technical Conference, p.181-194, June 10-15, 2002
• 21: Wright, C. P., Martino, M., and Zadok, E. 2003. NCryptfs: A secure and convenient cryptographic file system. In Proceedings of the Annual USENIX Technical Conference (San Antonio, Tex.), 197--210.
• 22: Zadok, E. and Nieh, J. 2000. FiST: A language for stackable file systems. In Proceedings of the Annual USENIX Technical Conference (San Diego, Calif.), 55--70.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management
          Subjects: File organization

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management
          Subjects: Directory structures

  E. Data
  E.5 FILES
      Subjects: Organization/structure

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.2 Information Storage
          Subjects: File organization

General Terms:
Design, Experimentation, Management

Keywords:
Namespace management, directory merging, snapshotting, stackable file systems, unification