File server scaling with network-attached secure disks

ABSTRACT

By providing direct data transfer between storage and client, network-attached storage devices have the potential to improve scalability for existing distributed file systems (by removing the server as a bottleneck) and bandwidth for new parallel and distributed file systems (through network striping and more efficient data paths). Together, these advantages influence a large enough fraction of the storage market to make commodity network-attached storage feasible. Realizing the technology's full potential requires careful consideration across a wide range of file system, networking and security issues. This paper contrasts two network-attached storage architectures---(1) Networked SCSI disks (NetSCSI) are network-attached storage devices with minimal changes from the familiar SCSI interface, while (2) Network-Attached Secure Disks (NASD) are drives that support independent client access to drive object services. To estimate the potential performance benefits of these architectures, we develop an analytic model and perform trace-driven replay experiments based on AFS and NFS traces. Our results suggest that NetSCSI can reduce file server load during a burst of NFS or AFS activity by about 30%. With the NASD architecture, server load (during burst activity) can be reduced by a factor of up to five for AFS and up to ten for NFS.

REFERENCES

• Anderson91: Thomas E. Anderson , Henry M. Levy , Brian N. Bershad , Edward D. Lazowska, The interaction of architecture and operating system design, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.108-120, April 08-11, 1991, Santa Clara, California, United States
• Anderson95: Anderson, D. (Seagate Technology), Personal communication, 1995.
• ANSI86: ANSI, "Small Computer System Interface (SCSI) Specification", ANSI X3.131-1986, 1986.
• ANSI95: ANSI, "SCSI-3 Fast-20 Parallel Interface", X3T10/i 047D Working Group, Revision 6.
• Arnould89: Emmanuel Arnould , H. T. Kung , Francois Bitz , Robert D. Sansom , Eric C. Cooperm, The design of nectar: a network backplane for heterogeneous multicomputers, Proceedings of the third international conference on Architectural support for programming languages and operating systems, p.205-216, April 03-06, 1989, Boston, Massachusetts, United States
• Baker91: Mary G. Baker , John H. Hartman , Michael D. Kupfer , Ken W. Shirriff , John K. Ousterhout, Measurements of a distributed file system, Proceedings of the thirteenth ACM symposium on Operating systems principles, p.198-212, October 13-16, 1991, Pacific Grove, California, United States
• Benner96: Alan F. Benner, Fibre channel: gigabit communications and I/O for computer networks, McGraw-Hill, Inc., Hightstown, NJ, 1995
• Berdahl95: Berdahl, L., Draft of "Parallel Transport Protocol Proposal", Lawrence Livermore National Labs, January 3, 1995.
• Birrell80: Birrell, A.D. and Needham, R.M., "A Universal File Server", IEEE Transactions on Software Engineering SE-6,5, Sept. 1980.
• Boden95: Nanette J. Boden , Danny Cohen , Robert E. Felderman , Alan E. Kulawik , Charles L. Seitz , Jakov N. Seizovic , Wen-King Su, Myrinet: A Gigabit-per-Second Local Area Network, IEEE Micro, v.15 n.1, p.29-36, February 1995 [doi>10.1109/40.342015]
• Brustoloni96: José Carlos Brustoloni , Peter Steenkiste, Effects of buffering semantics on I/O performance, Proceedings of the second USENIX symposium on Operating systems design and implementation, p.277-291, October 29-November 01, 1996, Seattle, Washington, United States
• Cao95: Pei Cao , Edward W. Felten , Anna R. Karlin , Kai Li, A study of integrated prefetching and caching strategies, Proceedings of the 1995 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, p.188-197, May 15-19, 1995, Ottawa, Ontario, Canada
• Chen90: Peter M. Chen , Garth A. Gibson , Randy H. Katz , David A. Patterson, An evaluation of redundant arrays of disks using an Amdahl 5890, Proceedings of the 1990 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.74-85, April 1990, Univ. of Colorado, Boulder, Colorado, United States
• Chen93: J. Bradley Chen , Brian N. Bershad, The impact of operating system structure on memory system performance, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.120-133, December 05-08, 1993, Asheville, North Carolina, United States
• Clark89: Clark, D.D. et al., "An Analysis of TCP Processing Overhead," IEEE Communications 27,6 (June 89), pp. 23-36.
• Cooper90: Cooper, E., et al., "Host Interface Design for ATM LANs", 16th Conference on Local Computer Networks, Oct. 199 I, pp. 247-258.
• Dahlin94: Dahlin, M. et al., "Cooperative Caching: Using Remote Client Memory to Improve File System Performance," First OSDI, pp. 267-280, Nov. 1994.
• Dahlin95: Dahlin, M.D. et al., "A Quantitative Analysis of Cache Policies for Scalable Network File Systems", 15th SOSE Dec. 1995.
• deJong93: Wiebren de Jonge , M. Frans Kaashoek , Wilson C. Hsieh, The logical disk: a new approach to improving file systems, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.15-28, December 05-08, 1993, Asheville, North Carolina, United States
• Drapeau94: A. L. Drapeau , K. W. Shirriff , J. H. Hartman , E. L. Miller , S. Seshan , R. H. Katz , K. Lutz , D. A. Patterson , E. K. Lee , P. M. Chen , G. A. Gibson, RAID-II: a high-bandwidth network file server, Proceedings of the 21ST annual international symposium on Computer architecture, p.234-244, April 18-21, 1994, Chicago, Illinois, United States
• Druschel93: Peter Druschel , Larry L. Peterson, Fbufs: a high-bandwidth cross-domain transfer facility, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.189-202, December 05-08, 1993, Asheville, North Carolina, United States
• Feeley95: M. J. Feeley , W. E. Morgan , E. P. Pighin , A. R. Karlin , H. M. Levy , C. A. Thekkath, Implementing global memory management in a workstation cluster, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.201-212, December 03-06, 1995, Copper Mountain, Colorado, United States
• Gibson92: Garth A. Gibson, Redundant disk arrays: reliable, parallel secondary storage, MIT Press, Cambridge, MA, 1992
• Gobioff96: Gobioff, H. et al., "Security for Network-Attached Storage Devices," CMU-CS-96-179, 1996.
• Golding95: Golding, R., et al., "Attribute-managed storage," Workshop on Modeling and Specification of I/O, San Antonio, TX, Oct. 1995.
• Grochowski96: Grochowski, E.G., Hoyt, R.F., "Future Trends in Hard Disk Drives," IEEE Transactions on Magnetics 32, 3 (May 1996), pp. 1850-1854.
• Hartman93: John H. Hartman , John K. Ousterhout, The Zebra striped network file system, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.29-43, December 05-08, 1993, Asheville, North Carolina, United States
• Hitz90: Hitz, D. et al., "Using UNIX as One Component of a Lightweight Distributed Kernel for Multiprocessor File Servers", Winter 1990 USENIX, pp. 285-295.
• Hitz94: Hitz, D., Lau, J. and Malcolm, M. "File Systems Design for an NFS File Server Appliance", Winter 1994 USENIX, Jan. 1994.
• Horst95: Robert W. Horst, TNet: A Reliable System Area Network, IEEE Micro, v.15 n.1, p.37-45, February 1995 [doi>10.1109/40.342016]
• Howard88: John H. Howard , Michael L. Kazar , Sherri G. Menees , David A. Nichols , M. Satyanarayanan , Robert N. Sidebotham , Michael J. West, Scale and performance in a distributed file system, ACM Transactions on Computer Systems (TOCS), v.6 n.1, p.51-81, Feb. 1988 [doi>10.1145/35037.35059]
• IEEE92: IEEE, "Scalable Coherent Interconnect", Standard 1596- 1992, 1992.
• IEEE94: IEEE P1244. "Reference Model for Open Storage Systems Interconnection-Mass Storage System Reference Model Version 5", Sept. 1995.
• Katz92: Katz, R.H., "High-Performance Network- and Channel- Attached Storage", Proceedings of the IEEE 80,8, Aug. 1992.
• Kim86: M Y Kim, Synchronized disk interleaving, IEEE Transactions on Computers, v.35 n.11, p.978-988, Nov. 1986 [doi>10.1109/TC.1986.1676699]
• Kronenberg86: Nancy P. Kronenberg , Henry M. Levy , William D. Strecker, VAXcluster: a closely-coupled distributed system, ACM Transactions on Computer Systems (TOCS), v.4 n.2, p.130-146, May 1986 [doi>10.1145/214419.214421]
• Lee95: E. K. Lee, Highly-available, scalable network storage, Proceedings of the 40th IEEE Computer Society International Conference, p.397, March 05-09, 1995
• Livny87: Miron Livny , Setrag Khoshafian , Haran Boral, Multi-disk management algorithms, Proceedings of the 1987 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.69-77, May 11-14, 1987, Banff, Alberta, Canada
• Long94: Darrell D. E. Long , Bruce R. Montague , Luis-Felipe Cabrera, Swift/RAID: a distributed RAID system, Computing Systems, v.7 n.3, p.333-359, Summer 1994
• Ma96: Qingming Ma , Peter Steenkiste , Hui Zhang, Routing high-bandwidth traffic in max-min fair share networks, Conference proceedings on Applications, technologies, architectures, and protocols for computer communications, p.206-217, August 28-30, 1996, Palo Alto, California, United States
• Maeda93: Chris Maeda , Brian N. Bershad, Protocol service decomposition for high-performance networking, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.244-255, December 05-08, 1993, Asheville, North Carolina, United States
• Massiglia94: Massiglia, P., ed., "The RAIDbook", RAID Advisory Board, 1994.
• Menascé96: Daniel A. Menascé , Odysseas I. Pentakalos , Yelena Yesha, An analytic model of hierarchical mass storage systems with network-attached storage devices, Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.180-189, May 23-26, 1996, Philadelphia, Pennsylvania, United States
• Miller88: S. W. Miller, A reference model for mass storage systems, Advances in computers, Academic Press Professional, Inc., San Diego, CA, 1988
• Minshall94: Minshall, G., Major, D., and Powell, K., "An Overview of the NetWare Operating System", Winter 1994 USENIX, 1994.
• National96: National Semiconductor. "The PersonaCard 100 Data Security Card," http://www.ipsecure.com/htm/persona.html.
• Nelson88: Michael N. Nelson , Brent B. Welch , John K. Ousterhout, Caching in the Sprite network file system, ACM Transactions on Computer Systems (TOCS), v.6 n.1, p.134-154, Feb. 1988 [doi>10.1145/35037.42183]
• NIST94: National Institute of Standards and Technology, "Digital Signature Standard" NIST FIPS Pub 186.
• NIST94a: National Institute of Standards and Technology, "Security Requirements for Cryptographic Modules", NIST FIPS 140-1.
• Ousterhout85: John K. Ousterhout , Hervé Da Costa , David Harrison , John A. Kunze , Mike Kupfer , James G. Thompson, A trace-driven analysis of the UNIX 4.2 BSD file system, Proceedings of the tenth ACM symposium on Operating systems principles, p.15-24, December 1985, Orcas Island, Washington, United States
• Ousterhout91: Ousterhout, J.K., "Why Aren't Operating Systems Getting Faster As Fast As Hardware?", Summer 1991 USENIX, June 1991, pp. 247-256.
• Patterson88: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• Patterson95: R. H. Patterson , G. A. Gibson , E. Ginting , D. Stodolsky , J. Zelenka, Informed prefetching and caching, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.79-95, December 03-06, 1995, Copper Mountain, Colorado, United States
• Rambus92: Rambus Inc., "Rambus Architectural Overview", 1992. http://www.rambus.com.
• Riedel96: Riedel, E. and Gibson, G. "Understanding Customer Dissatisfaction With Underutilized Distributed File Servers", 5th Goddard Conference on Mass Storage Systems and Technologies, College Park, MD, Sept. 1996.
• Ruemmler91: Ruemmler, C. and Wilkes, J., "Disk Shuffling", Hewlett-Packard Laboratories Concurrent Systems Project Tech Report HPL-CSP-91-30.
• Sachs94: Martin W. Sachs , Avraham Leff , Denise Sevigny, LAN and I/O Convergence: A Survey of the Issues, Computer, v.27 n.12, p.24-32, December 1994 [doi>10.1109/2.335725]
• Sandberg85: Sandberg, R. et al., "Design and Implementation of the Sun Network Filesystem", Summer 1985 USENIX, June 1985, pp. 119-130.
• Seagate96: Seagate Corporation, "Barracuda Family Product Brief (ST19171)", 1996.
• Siu95: Kai-Yeung Siu , Raj Jain, A brief overview of ATM: protocol layers, LAN emulation, and traffic management, ACM SIGCOMM Computer Communication Review, v.25 n.2, p.6-20, April 1995 [doi>10.1145/210613.210616]
• Srivastava94: Srivastava, A., and Eustace, A., "ATOM: A system for building customized program analysis tools", WRL Techmcal Report TN-41, 1994.
• StorageTek94: Storage Technology Corporation, "Iceberg 9200 Storage System: Introduction", STK Part Number 307406101, Storage Technology Corporation, 1994.
• Steenkiste94: Peter A. Steenkiste, A systematic approach to host interface design for high-speed networks, Computer, v.27 n.3, p.47-57, March 1994 [doi>10.1109/2.268886]
• Traw95: Traw, C.B.S. and Smith, J.M., "Striping Within the Network Subsystem", IEEE Network, Jul./Aug. 1995.
• Tygar95: Tygar, J.D., and Yee, B.S., "Secure Coprocessors in Electronic Commerce Applications," 1995 USENIX Electronic Commerce Workshop, 1995, New York.
• VanMeter96: Rodney Van Meter, A brief survey of current work on network attached peripherals (extended abstract), ACM SIGOPS Operating Systems Review, v.30 n.1, p.63-70, Jan. 1996 [doi>10.1145/218646.218650]
• VanMeter96a: Van Meter, R., Holtz, S., and Finn G., "Derived Virtual Devices: A Secure Distributed File System Mechanism", 5th Goddard Conference on Mass Storage Systems and Technologies", College Park, MD, Sept. 1996.
• Varma95: Anujan Varma , Quinn Jacobson, Destage algorithms for disk arrays with non-volatile caches, Proceedings of the 22nd annual international symposium on Computer architecture, p.83-95, June 22-24, 1995, S. Margherita Ligure, Italy
• vonEicken92: Thorsten von Eicken , David E. Culler , Seth Copen Goldstein , Klaus Erik Schauser, Active messages: a mechanism for integrated communication and computation, Proceedings of the 19th annual international symposium on Computer architecture, p.256-266, May 19-21, 1992, Queensland, Australia
• Watson95: R. W. Watson , R. A. Coyne, The parallel I/O architecture of the high-performance storage system (HPSS), Proceedings of the 14th IEEE Symposium on Mass Storage Systems, p.27, September 11-14, 1995
• Weingart87: Weingart, S.H., "Physical Security of the I.tABYSS System", IEEE Computer Society Conference on Security and Privacy, 1987, pp. 52-58.
• White87: White, S.R. and Comerford, L., "ABYSS: A Trusted Architecture for Software Protection", IEEE Computer Society Conference on Security and Privacy, 1987, pp. 38-51.
• Wilkes95: J. Wilkes , R. Golding , C. Staelin , T. Sullivan, The HP AutoRAID hierarchical storage system, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.96-108, December 03-06, 1995, Copper Mountain, Colorado, United States
• Wiltzius95: Wiltzius, D. et al., "Network-attached peripherals for HPSS/SIOF", http://www.llnl.gov/liv_comp/siof/siof_nap.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management
          Subjects: Distributed file systems

General Terms:
Design, Experimentation, Measurement, Performance, Theory