Extending stability beyond CPU millennium: a micron-scale atomistic simulation of Kelvin-Helmholtz instability

ABSTRACT

We report the computational advances that have enabled the first micron-scale simulation of a Kelvin-Helmholtz (KH) instability using molecular dynamics (MD). The advances are in three key areas for massively parallel computation such as on BlueGene/L (BG/L): fault tolerance, application kernel optimization, and highly efficient parallel I/O. In particular, we have developed novel capabilities for handling hardware parity errors and improving the speed of interatomic force calculations, while achieving near optimal I/O speeds on BG/L, allowing us to achieve excellent scalability and improve overall application performance. As a result we have successfully conducted a 2-billion atom KH simulation amounting to 2.8 CPU-millennia of run time, including a single, continuous simulation run in excess of 1.5 CPU-millennia. We have also conducted 9-billion and 62.5-billion atom KH simulations. The current optimized ddcMD code is benchmarked at 115.1 TFlop/s in our scaling study and 103.9 TFlop/s in a sustained science run, with additional improvements ongoing. These improvements enabled us to run the first MD simulations of micron-scale systems developing the KH instability.

REFERENCES

• 1: M. P. Allen , D. J. Tildesley, Computer simulation of liquids, Clarendon Press, New York, NY, 1988
• 2: S. Chandrasekhar. Hydrodynamic and Hydromagnetic Stability. Oxford Univ. Press, Oxford, 1961.
• 3: M. P. Ciamarra, A. Coniglio, and M. Micodemi. Shear instabilities in granular mixtures. Phys. Rev. Lett., 94:188001, 2005.
• 4: P. Crozier, F. Reid, and C. Vaughn. Lammps benchmarks, http://lammps.sandia.gov/bench.html.
• 5: M. S. Daw and M. I. Baskes. Embedded-atom method: Derivation and application to impurities, surfaces, and other defects in metals. Phys. Rev. B, 29(12):6443--6453, 1984.
• 6: P. G. Drazin and W. H. Reid. Hydrodynamic Stability. Cambridge Univ. Press, Cambridge, 1981.
• 7: E. N. (Mootaz) Elnozahy , Lorenzo Alvisi , Yi-Min Wang , David B. Johnson, A survey of rollback-recovery protocols in message-passing systems, ACM Computing Surveys (CSUR), v.34 n.3, p.375-408, September 2002 [doi>10.1145/568522.568525]
• 8: C. Engelmann and A. Geist. Super-scalable algorithms for computing on 100,000 processors. In Computational Science - ICCS 2005, PT 1, Proceedings, volume 3514 of Lecture Notes in Computer Science, pages 313--321, 2005.
• 9: O. B. Fringer and R. L. Street. The dynamics of breaking progressive interfacial waves. J. Fluid Mech., 494:319--353, 2003.
• 10: D. C. Fritts, C. Bizon, J. A. Werne, and C. K. Meyer. Layering accompanying turbulence generation due to shear instability and gravity-wave breaking. J. Geophys. Res., 108:20:1--13, 2003.
• 11: D. C. Fritts, T. L. Palmer, O. Andreassen, and I. Lie. Evolution and breakdown of kelvin-helmholtz billows in stratified compressible flows:1, comparison of two-and three-dimensional flows. J. Atmos. Sci., 53:3173--3191, 1996.
• 12: T. C. Germann, K. Kadau, and P. S. Lomdahl. 25 tflop/s multibillion-atom molecular dynamics simulations and visualization/analysis on bluegene/1. Proc. Supercomputing 2005, Seattle, 2005, on CD-ROM, http://sc05.supercomputing.org/schedule/pdf/papl22.pdf.
• 13: Mohamed Gomaa , Chad Scarbrough , T. N. Vijaykumar , Irith Pomeranz, Transient-fault recovery for chip multiprocessors, Proceedings of the 30th annual international symposium on Computer architecture, June 09-11, 2003, San Diego, California
• 14: Grape newletter vol.8 (dec 16 2006), http://grape.mtk.nao.ac.jp/grape/newsletter/061216.html.
• 15: J. F. Hawley and N. J. Zabusky. Vortex paradigm for shock-accelerated density-stratified interfaces. Phys. Rev. Lett., 63(12): 1241--1244, 1989.
• 16: J. K. Holt, H. G. Park, Y. Wang, M. Stadermann, A. B. Artyukhin, C. P. Grigoropoulos, A. Noy, and O. Bakajin. Fast Mass Transport Through Sub-2-Nanometer Carbon Nanotubes. Science, 312(5776):1034--1037, 2006.
• 17: R. A. Johnson. Alloy models with the embedded-atom method. Phys. Rev. B, 39:12554, 1989.
• 18: R. A. Johnson and D. J. Oh. Analytic embedded atom method model for bcc metals. J. Mater. Res., 4:1195, 1989.
• 19: V. Junk, F. Heitsch, and T. Naab. The kelvin-helmholtz instability in smoothed particle hydrodynamics. Proc. Int. Astro. Union, 2:210--210, 2007.
• 20: K. Kadau, T. C. Germann, N. G. Hadjiconstantinou, P. S. Lomdahl, G. Dimonte, B. L. Holian, and B. J. Alder. Nanohydrodynamics simulations: An atomistic view of the rayleigh-taylor instability. Proc. Natl. Acad. Sci. USA, 101:5851--5855, 2004.
• 21: K. Kadau, T. C. Germann, and P. S. Lamdahl. Molecular dynamics comes of age: 320 billion atom simulation on bluegene/1. Int. J. Mod. Phys. C, 17:1755, 2006.
• 22: H. Lamb. Hydrodynamics. Cambridge Univ. Press, Cambridge, 6th edition, 1932.
• 23: D. R. Mason, R. E. Rudd, and A. P. Sutton. Stochastic kinetic monte carlo algorithms for long-range hamiltonians. Computer Physics Comm., 160:140--157, 2004.
• 24: J. J. Monaghan. Flaws in the modern laplacian theory. Earth, Moon and Planets, 71:73--84, 1995.
• 25: J. A. Moriarty. Analytic representation of multi-ion interatomic potentials in transition metals. Phys. Rev. B, 42:1609--1628, 1990.
• 26: J. A. Moriarty. Angular forces and melting in bcc transition metals: A case study of molybdenum. Phys. Rev. B, 49:12431--12445, 1994.
• 27: Tetsu Narumi , Yousuke Ohno , Noriaki Okimoto , Takahiro Koishi , Atsushi Suenaga , Noriyuki Futatsugi , Ryoko Yanai , Ryutaro Himeno , Shigenori Fujikawa , Makoto Taiji , Mitsuru Ikei, A 55 TFLOPS simulation of amyloid-forming peptides from yeast prion Sup35 with the special-purpose computer system MDGRAPE-3, Proceedings of the 2006 ACM/IEEE conference on Supercomputing, November 11-17, 2006, Tampa, Florida [doi>10.1145/1188455.1188506]
• 28: T. L. Palmer, D. C. Fritts, and O. Andreassen. Evolution and breakdown of kelvin-helmholtz billows in stratified compressible flows:2. instability structure, evolution, and energetics. J. Atmos. Sci., 53:3192--3212, 1996.
• 29: D. C. Rapaport, The Art of Molecular Dynamics Simulation, Cambridge University Press, New York, NY, 1995
• 30: http://www.sandia.gov/asc/redstorm.html.
• 31: George A. Reis , Jonathan Chang , Neil Vachharajani , Ram Rangan , David I. August, SWIFT: Software Implemented Fault Tolerance, Proceedings of the international symposium on Code generation and optimization, p.243-254, March 20-23, 2005 [doi>10.1109/CGO.2005.34]
• 32: George A. Reis , Jonathan Chang , Neil Vachharajani , Ram Rangan , David I. August , Shubhendu S. Mukherjee, Design and Evaluation of Hybrid Fault-Detection Systems, ACM SIGARCH Computer Architecture News, v.33 n.2, p.148-159, May 2005
• 33: R. E. Rudd and J. Q. Broughton. Coupling of length scales in solid state systems. Phys. Stat. Sol., 217:251--291, 2000.
• 34: SETI@HOME project: http://setiathome.ssl.berkeley.edu.
• 35: F. H. Streitz, J. N. Glosli, and M. V. Patel. Beyond finite-size scaling in solidification simulations. Phys. Rev. Lett., 96:225701, 2006.
• 36: F. H. Streitz, J. N. Glosli, and M. V. Patel. Simulating solidification in metals at high pressure: The drive to petascale computing. J. Phys.: Conf. Ser., 46:254--267, 2006.
• 37: R. I. Sykes and W. S. Lewellen. A numerical study of breaking of kelvin-helmholtz billows using a reynolds-stress turbulence closure model. J. Atmos. Sci., 39:1506--1520, 1982.
• 38: TOP500 Supercomputer Sites: http://www.top500.org/.
• 39: J.-Y. Yang and J.-W. Chang. Rarefied flow instability simulation using model boltzmann equations. Proc. Fluid Dynamics Conf. 28th, Snowmass Village, CO, June 29-July 2, 1997.
• 40: Y. Yeh. Triple-triple redundant 777 primary flight computer. In Proceedings of the 1996 IEEE Aerospace Applications Conference, volume 1, pages 293--307, 1996.
• 41: R. Zhang, X. He, G. Doolen, and S. Chen. Surface tension effect on two-dimensional two-phase kelvin-helmholtz instabilities. Adv. Water Res., 24:461--478, 2001.