Taming the storage dragon: the adventures of hoTMaN

ABSTRACT

HoTMaN (HoT-standby MaNager) is a joint project between MySpace and USC Database Laboratory to design and develop a tool to ensure a 24x7 up-time and ease administration of Terabytes of storage that sits underneath hundreds of database servers. The HoTMaN tool's innovation and uniqueness is that it can, with a few clicks, perform operational tasks that require hundreds of keyboard strokes by "trusted trained" experts. With HoTMaN, MySpace can within minutes migrate the relational database(s) of a failed server to a hot-standby. A process that could take over 1 hour and had a high potential for human error is now performed reliably. A database internal to HoTMaN captures all virtual disks, volume and file configurations associated with each SQL Server and candidate hot-standby servers where SQL server processing could be migrated. HoTMaN is deployed in production and its current operational benefits include: (i) enhanced availability of data, and (ii) planned maintenance and patching.

REFERENCES

• 1: Mache Creeger, CTO Storage Roundtable: Part II, Communications of the ACM, v.51 n.9, September 2008 [doi>10.1145/1378727.1378741]
• 2: D. J. Dewitt , S. Ghandeharizadeh , D. A. Schneider , A. Bricker , H. -I. Hsiao , R. Rasmussen, The Gamma Database Machine Project, IEEE Transactions on Knowledge and Data Engineering, v.2 n.1, p.44-62, March 1990 [doi>10.1109/69.50905]
• 3: Shahram Ghandeharizadeh , David J. DeWitt, Hybrid-range partitioning strategy: a new declustering strategy for multiprocessor databases machines, Proceedings of the sixteenth international conference on Very large databases, p.481-492, September 1990, Brisbane, Australia
• 4: S. Ghandeharizadeh, S. Gao, C. Gahagan, and R. Krauss. An On-Line Re-Organization Framework for SAN File Systems. In ADBIS, volume 4152 of Lecture Notes in Computer Science, pages 399--414. Springer, 2006.
• 5: Miron Livny , Setrag Khoshafian , Haran Boral, Multi-disk management algorithms, Proceedings of the 1987 ACM SIGMETRICS conference on Measurement and modeling of computer systems, p.69-77, May 11-14, 1987, Banff, Alberta, Canada
• 6: Nagios. http://www.nagios.org. 2008.
• 7: M.A. Olson, K. Bostic, and M.I. Seltzer. Berkeley. In USENIX Annual Technical Conference, FREENIX Track, pages 183--191, 1999.
• 8: Michael Stonebraker , Samuel Madden , Daniel J. Abadi , Stavros Harizopoulos , Nabil Hachem , Pat Helland, The end of an architectural era: (it's time for a complete rewrite), Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 9: A. Wergeles. HoTMaN Discussions on October 17, 2008. Private Communication, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Parallel databases

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability
          Subjects: Fault-tolerance

General Terms:
Management

Keywords:
data availability, storage area networks