Blue matter: approaching the limits of concurrency for classical molecular dynamics

ABSTRACT

This paper describes a novel spatial-force decomposition for N-body simulations for which we observe O(sqrt(p)) communication scaling. This has enabled Blue Matter to approach the effective limits of concurrency for molecular dynamics using particle-mesh (FFT-based) methods for handling electrostatic interactions. Using this decomposition, Blue Matter running on Blue Gene/L has achieved simulation rates in excess of 1000 time steps per second and demonstrated significant speed-ups to O(1) atom per node. Blue Matter employs a Communicating Sequential Process (CSP) style model with application communication state machines compiled to hardware interfaces. The scalability achieved has enabled methodologically rigorous biomolecular simulations on biologically interesting systems, such as membrane-bound proteins, whose time scales dwarf previous work on those systems, Major scaling improvements will require exploration of alternative algorithms for treating the long range electrostatics.

REFERENCES

• 1: F. Allen , G. Almasi , W. Andreoni , D. Beece , B. J. Berne , A. Bright , J. Brunheroto , C. Cascaval , J. Castanos , P. Coteus , P. Crumley , A. Curioni , M. Denneau , W. Donath , M. Eleftheriou , B. Fitch , B. Fleischer , C. J. Georgiou , R. Germain , M. Giampapa , D. Gresh , M. Gupta , R. Haring , H. Ho , P. Hochschild , S. Hummel , T. Jonas , D. Lieber , G. Martyna , K. Maturu , J. Moreira , D. Newns , M. Newton , R. Philhower , T. Picunko , J. Pitera , M. Pitman , R. Rand , A. Royyuru , V. Salapura , A. Sanomiya , R. Shah , Y. Sham , S. Singh , M. Snir , F. Suits , R. Swetz , W. C. Swope , N. Vishnumurthy , T. J. C. Ward , H. Warren , R. Zhou, Blue Gene: a vision for protein science using a petaflop supercomputer, IBM Systems Journal, v.40 n.2, p.310-327, February 2001
• 2: Deserno, M., and Holm, C. 1998. How to mesh up Ewald sums. i. a theoretical and numerical comparison of various particle mesh routines. J. Chem. Phys. 109, 18, 7678--7693.
• 3: Eleftheriou, M., Fitch, B., Rayshubskiy, A., Ward, T., and Germain, R. 2005. Performance measurements of the 3d FFT on the Blue Gene/L supercomputer. In Euro-Par 2005 Parallel Processing: 11th International Euro-Par Conference, Lisbon, Portugal, August 30-September 2, 2005. Springer-Verlag, J. Cunha and P. Medeiros, Eds., vol. 3648 of Lecture Notes in Computer Science, 795--803.
• 4: Eleftheriou, M., Germain, R., Royyuru, A., and Zhou, R. 2006. Thermal denaturing of mutant lysozyme with both oplsaa and charmm force fields. submitted to J. Am. Chem. Soc.
• 5: Eleftheriou, M., Rayshubskiy, A., Pitera, J. W., Fitch, B. G., Zhou, R., and Germain, R. S. 2006. Parallel implementation of the replica exchange molecular dynamics algorithm on Blue Gene/L. In Fifth IEEE International Workshop on High Performance Computational Biology.
• 6: B. G. Fitch , R. S. Germain , M. Mendell , J. Pitera , M. Pitman , A. Rayshubskiy , Y. Sham , F. Suits , W. Swope , T. J. C. Ward , Y. Zhestkov , R. Zhou, Blue Matter, an application framework for molecular simulation on blue gene, Journal of Parallel and Distributed Computing, v.63 n.7-8, p.759-773, July 2003 [doi>10.1016/S0743-7315(03)00084-4]
• 7: Fitch, B. G., Rayshubskiy, A., Eleftheriou, M., Ward, T. C., Giampapa, M., Zhestkov, Y., Pitman, M. C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Germain, R. S., and Feller, S. 2005. Blue matter: Strong scaling of molecular dynamics on Blue Gene/L. Research Report RC23688, IBM Research Division, August.
• 8: Fitch, B. G., Rayshubskiy, A., Eleftheriou, M., Ward, T. C., Giampapa, M., Zhestkov, Y., Pitman, M. C., Suits, F., Grossfield, A., Pitera, J., Swope, W., Zhou, R., Feller, S., and Germain, R. S. 2006. Blue Matter: Strong scaling of molecular dynamics on Blue Gene/L. In International Conference on Computational Science (ICCS 2006), Springer-Verlag, V. Alexandrov, D. van Albada, P. Sloot, and J. Dongarra, Eds., vol. 3992 of LNCS, 846--854.
• 9: Dan Frenkel , Daan Frenkel , Berend Smit, Understanding Molecular Simulation, Academic Press, Inc., Orlando, FL, 2001
• 10: Gara, A., ET AL. 2005. Overview of the Blue Gene/L system architecture. IBM Journal of Research and Development 49, 2/3, 195--212.
• 11: Robert S. Germain , Blake Fitch , Aleksandr Rayshubskiy , Maria Eleftheriou , Michael C. Pitman , Frank Suits , Mark Giampapa , T.J. Christopher Ward, Blue matter on blue gene/L: massively parallel computation for biomolecular simulation, Proceedings of the 3rd IEEE/ACM/IFIP international conference on Hardware/software codesign and system synthesis, September 19-21, 2005, Jersey City, NJ, USA [doi>10.1145/1084834.1084888]
• 12: Germain, R., Zhestkov, Y., Eleftheriou, M., Rayshubskiy, A., Suits, F., Ward, T., and Fitch, B. 2005. Early performance data on the Blue Matter molecular simulation framework. IBM Journal of Research and Development 49, 2/3, 447--456.
• 13: Giampapa, M., Bellofatto, R., Blumrich, M. A., Chen, D., Dombrowa, M. B., Gara, A., Haring, R. A., Heidelberger, P., Hoenicke, D., Kopcsay, G. V., Nathanson, B. J., Steinmacher-Burow, B. D., Ohmacht, M., Salapura, V., and Vranas, P. 2005, Blue Gene/L advanced diagnostics environment. IBM Journal of Research and Development 49, 2/3, 319--332.
• 14: Grossfield, A., Feller, S. E., and Pitman, M. C. 2006. A role for direct interactions in the modulation of rhodopsin by omega-3 polyunsaturated lipids. PNAS 103, 13, 4888--4893.
• 15: Bruce Hendrickson , Steve Plimpton, Parallel many-body simulations without all-to-all communication, Journal of Parallel and Distributed Computing, v.27 n.1, p.15-25, May 1995 [doi>10.1006/jpdc.1995.1068 ]
• 16: C. A. R. Hoare, Communicating sequential processes, Prentice-Hall, Inc., Upper Saddle River, NJ, 1985
• 17: Kumar, S., Huang, C., Almasi, G., and Kale, L. V. 2006. Achieving strong scaling with NAMD on Blue Gene/l. 20th IEEE International Parallel and Distributed Processing Symposium, IEEE. http://charm.cs.uiuc.edu/papers/NAMDIDPDSO6.pdf.
• 18: James C. Phillips , Gengbin Zheng , Sameer Kumar , Laxmikant V. Kalé, NAMD: biomolecular simulation on thousands of processors, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-18, November 16, 2002, Baltimore, Maryland
• 19: Pitman, M. C., Suits, F., MacKerell, Alexander D., J., and Feller, S. E. 2004. Molecular-level organization of saturated and polyunsaturated fatty acids in a phosphatidylcholine bilayer containing cholesterol. Biochemistry 43, 49, 15318--15328.
• 20: Pitman, M. C., Grossfield, A., Suits, F., and Feller, S. E. 2005. Role of cholesterol and polyunsaturated chains in lipid-protein interactions: Molecular dynamics simulation of rhodopsin in a realistic membrane environment. Journal of the American Chemical Society 127, 13, 4576--4577.
• 21: Pitman, M. C., Suits, F., Gawrisch, K., and Feller, S. E. 2005. Molecular dynamics investigation of dynamical properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122, 24, 244715.
• 22: Plimpton, S., and Hendrickson, B. 1996. A new parallel method for molecular dynamics simulation of macromolecular systems. Journal of Computational Chemistry 17, 3, 326--337.
• 23: Shaw, D. 2004. An asymptotic improvement in the parallel evaluation of pairwise particle interactions. Presented at Philadelpha American Chemical Society meeting, September.
• 24: Shaw, D. E. 2005. A fast, scalable method for the parallel evaluation of distance-limited pairwise particle interactions. Journal of Computational Chemistry 26, 13, 1318--1328.
• 25: Snir, M. 2004. A note on n-body computations with cutoffs. Theory of Computing Systems 37, 295--318. DOI: 10.1007/s00224-003-1071-0.
• 26: Sugita, Y., and Okamoto, Y. 1999. Replica-exchange molecular dynamics method for protein folding. Chem. Phys. Lett. 314, 141--151.
• 27: Suits, F., Pitman, M. C., and Feller, S. E. 2005. Molecular dynamics investigation of the structural properties of phosphatidylethanolamine lipid bilayers. Journal of Chemical Physics 122, 24.
• 28: Swope, W., Andersen, H., Berens, P., and Wilson, K. 1982. A computer simulation method for the calculation of equilibrium constants for the formation of physical clusters of molecules: Application to small water clusters. Journal of Chemical Physics 76, 637--649.
• 29: Swope, W. C., Pitera, J. W., Suits, F., Pitman, M., Eleftheriou, M., Fitch, B. G., Germain, R. S., Rayshubskiy, A., Ward, T. J. C., Zhestkov, Y., and Zhou, R. 2004. Describing protein folding kinetics by molecular dynamics simulations. 2. example applications to alanine dipeptide and a β-hairpin peptide. J. Phys. Chem. B 108, 21, 6582--6594.
• 30: Valerie E. Taylor , Rick L. Stevens , Kathryn E. Arnold, Parallel molecular dynamics: implications for massively parallel machines, Journal of Parallel and Distributed Computing, v.45 n.2, p.166-175, Sept. 15, 1997 [doi>10.1006/jpdc.1997.1370 ]

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.0 General

Additional Classification:
  D. Software
  D.1 PROGRAMMING TECHNIQUES
      D.1.3 Concurrent Programming
          Subjects: Parallel programming

  G. Mathematics of Computing
  G.4 MATHEMATICAL SOFTWARE
      Subjects: Parallel and vector implementations

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Parallel

General Terms:
Algorithms, Design, Theory

Keywords:
molecular dynamics, n-body simulations, parallel programming