Terascale spectral element dynamical core for atmospheric general circulation models

ABSTRACT

Climate modeling is a grand challenge problem where scientific progress is measured not in terms of the largest problem that can be solved but by the highest achievable integration rate. These models have been notably absent in previous Gordon Bell competitions due to their inability to scale to large processor counts. A scalable and efficient spectral element atmospheric model is presented. A new semi-implicit time stepping scheme accelerates the integration rate relative to an explicit model by a factor of two, achieving 130 years per day at T63L30 equivalent resolution. Execution rates are reported for the standard shallow water and Held-Suarez climate benchmarks on IBM SP clusters. The explicit T170 equivalent multi-layer shallow water model sustains 343 Gflops at NERSC, 206 Gflops at NPACI (SDSC) and 127 Gflops at NCAR. An explicit Held-Suarez integration sustains 369 Gflops on 128 16-way IBM nodes at NERSC.

REFERENCES

• 1: Boer, G. J., and B. Denis, 1997: Numerical convergence of the dynamics of a GCM. Climate Dynamics,13, 359-374.
• 2: Eduardo D''Azevedo , Victor Eijkhout , Charles Romine, LAPACK Working Note 56: Reducing Communication Costs in the Conjugate Gradient Algorithm on Distributed Memory Multiprocessors, University of Tennessee, Knoxville, TN, 1993
• 3: Held, I. H., and M. J. Suarez, 1994: A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models. Bull. Amer. Met. Soc.,75, 1825-1830.
• 4: IBM SC-09-2718-00, 1998: XL Fortran for AIX. Language Reference. First edition. Version 6, Release 1, p. 746.
• 5: Iskandarani, M., D. B. Haidvogel, and J. P. Boyd, 1995: A staggered spectral element model with application to the oceanic shallow water equations. Int. J. Numer. Meth. Fluids,20, 394-414.
• 6: Ruediger Jakob-Chien , James J. Hack , David L. Williamson, Spectral transform solutions to the shallow water test set, Journal of Computational Physics, v.119 n.1, p.164-187, June 1995 [doi>10.1006/jcph.1995.1125 ]
• 7: Karniadakis, G. M., and S. J. Sherwin, 1999: Spectral/hp Element Methods for CFD. Oxford University Press, Oxford, England, 390p.
• 8: George Karypis , Vipin Kumar, Multilevel k-way partitioning scheme for irregular graphs, Journal of Parallel and Distributed Computing, v.48 n.1, p.96-129, Jan. 1998 [doi>10.1006/jpdc.1997.1404 ]
• 9: Rancic, M., R. J. Purser, and F. Mesinger, 1996: A global shallow-water model using an expanded spherical cube: Gnomic versus conformal coordinates. Q. J. R. Meteorol. Soc.,122, 959-982.
• 10: Ritchie, H., C. Temperton, A. Simmons, M. Hortal, T. Davies, D. Dent, and M. Hamrud. Implementation of the semi-Lagrangian method in a high-resolution version of the ECMWF forecast model. Mon. Wea. Rev.,123, 489-514.
• 11: Ronquist, E. M., 1988: Optimal Spectral Element Methods for the Unsteady Three Dimensional Navier Stokes Equations, Ph.D Thesis, Massachusetts Institute of Technology, 176p.
• 12: Sadourny, R., 1972: Conservative finite-difference approximations of the primitive equations on quasi-uniform spherical grids. Mon. Wea. Rev.,100, 136-144.
• 13: Simmons, A. J., and D. M. Burridge, 1981: An energy and angular-momentum conserving vertical finite-difference scheme and hybrid vertical coordinates. Mon. Wea. Rev.,109, 758-766.
• 14: Mark Taylor , Joseph Tribbia , Mohamed Iskandarani, The spectral element method for the shallow water equations on the sphere, Journal of Computational Physics, v.130 n.1, p.92-108, Jan. 1, 1997 [doi>10.1006/jcph.1996.5554 ]
• 15: S. J. Thomas , R. D. Loft, Parallel Semi-Implicit Spectral Element Methods for Atmospheric General Circulation Models, Journal of Scientific Computing, v.15 n.4, p.499-518, December 2000 [doi>10.1023/A:1011188832645 ]
• 16: H. M. Tufo , P. F. Fischer, Terascale spectral element algorithms and implementations, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.68-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331599]
• 17: Williamson, D. L., J. B. Drake, J. J. Hack, R. Jakob, and P. N. Swarztrauber, 1992: A standard test set for numerical approximations to the shallow water equations in spherical geometry. J. Comp. Phys.,102, 211-224.
• 18: Worley P. H., and I. T. Foster, 1994: Parallel Spectral Transform Shallow Water Model: A Runtime-tunable parallel benchmark code. Proceedings of the Scalable High Performance Computing Conference, eds. J. J. Dongarra and D. W. Walker, IEEE Computer Society Press, 207-214.
• 19: Worley P. H., 2001: Personal communication.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.1 Large and Medium ("Mainframe") Computers
          Subjects: Super (very large) computers

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Combined

  J. Computer Applications
  J.2 PHYSICAL SCIENCES AND ENGINEERING
      Subjects: Earth and atmospheric sciences

General Terms:
Algorithms, Design, Performance