Y. Cui
ABSTRACT
We have developed a highly scalable and efficient GPU-based finite-difference code (AWP) for earthquake simulation that implements high throughput, memory locality, communication reduction and communication/computation overlap and achieves linear scalability on Cray XK7 Titan at ORNL and NCSA's Blue Waters system. We simulate realistic 0-10 Hz earthquake ground motions relevant to building engineering design using high-performance AWP. Moreover, we show that AWP provides a speedup by a factor of 110 in key strain tensor calculations critical to probabilistic seismic hazard analysis (PSHA). These performance improvements to critical scientific application software, coupled with improved co-scheduling capabilities of our workflow-managed systems, make a statewide hazard model a goal reachable with existing supercomputers. The performance improvements of GPU-based AWP are expected to save millions of core-hours over the next few years as physics-based seismic hazard analysis is developed using heterogeneous petascale supercomputers.
- Bielak, J., Graves, R., Olsen, K. B., Taborda, R., Ramirez-Guzman, L., Day, S., Ely, G., Roten, D., Jordan, T., Maechling, P., Urbanic, J., Cui, Y. and Juve, G. 2010. The ShakeOut earthquake scenario: Verification of three simulation sets. Geophysical Journal International., 180, 1 (Jan. 2010), 375--404.Google Scholar
Cross Ref
- Blanch, J. O., Robertsson, J. O. and Symes, W. W. 1995. Modeling of a constant Q: methodology and algorithm for an efficient and optimally inexpensive viscoelastic technique. Geophysics, 60, 1, 176--184.Google ScholarCross Ref
- Callaghan, S., Deelman, E., Gunter, D., Juve, G., Maechling, P., Brooks, C., Vahi, K., Milner, K., Graves, R., Field, E., Okaya, D. and Jordan, T. 2010. Scaling up workflow-based applications. Journal of Computer and System Sciences, 76, 6 (Sep. 2010), 428--446. Google ScholarDigital Library
- Cerjan, C., Kosloff, D., Kosloff, R. and Reshef, M. 1985. A nonreflecting boundary condition for discrete acoustic and elastic wave equations. Geophysics, 50, 4, 705--708.Google ScholarCross Ref
- Chen, P., Jordan, T. H. and Zhao, L. 2007. Full three-dimensional tomography: a comparison between the scattering- integral and adjoint-wavefield methods. Geophysical Journal International., 170, 1, 175--181.Google ScholarCross Ref
- Chen, P., Zhao, L. and Jordan, T. H. 2007, Full 3D tomography for the crustal structure of the Los Angeles region. Bulletin of the Seismological Society of America, 97, 4, 1094--1120.Google ScholarCross Ref
- Cui, Y., Olsen, K. B., Jordan, T. H., Lee, K., Zhou, J., Small, P., Roten, D., Ely, G., Panda, D. K., Chourasia, A., Levesque, J., Day, S. M. and Maechling, P. 2010. Scalable earthquake simulation on petascale supercomputers. Proc. of Int'l. Conf. for High Performance Computing, Networking, Storage and Analysis (SC'10, New Orleans, November 2010), 1--20. Google ScholarDigital Library
- Dalguer, L. A. and Day, S. M. 2007. Staggered-grid split-node method for spontaneous rupture simulation. Journal of Geophysical Research: Solid Earth, 112, B02302. doi:10.1029/2006JB004467.Google ScholarCross Ref
- Day, S. M. 1998. Efficient simulation of constant Q using coarse-grained memory variables. Bulletin of the Seismological Society of America, 88, 4, 1051--1062.Google Scholar
- Day, S. M. and Bradley, C. R. 2001. Memory-efficient simulation of anelastic wave propagation. Bulletin of the Seismological Society of America, 91, 3, 520--531.Google ScholarCross Ref
- Field, E. H., Dawson, T. E., Felzer, K. R., Frankel, A. D., Gupta, V., Jordan, T. H., Parsons, T., Petersen, M. D., Stein, R. S., Weldon II, R. J. and Wills, C. J. 2009. Uniform california earthquake rupture forecast, version 2 (UCERF 2). Bulletin of the Seismological Society of America, vol. 99, no. 4 (Aug. 2009), 2053--2107.Google Scholar
- Field, E. H., Jordan, T. H. and Cornell, C. A. 2003. OpenSHA: A developing community-modeling environment for seismic hazard analysis. Seismological Research Letters, 74, 4, 406--419.Google ScholarCross Ref
- Georgia Tech 2013. Keeneland User Guide. {Online}. https://www.xsede.org/gatech-keeneland.Google Scholar
- Graves, R. W. 1996. Simulating seismic wave propagation in 3D elastic media using staggered-grid finite differences. Bulletin of the Seismological Society of America, 86, 4, 1091--1106.Google Scholar
- Graves, R. W. and Pitarka, A. 2010. Broadband ground-motion simulation using a hybrid approach. Bulletin of the Seismological Society of America, 100, 5A, 2095--2123.Google ScholarCross Ref
- Graves, R., Aagaard, B., Hudnut, K., Star, L., Stewart, J. and Jordan, T. H. 2008. Broadband simulations for Mw 7.8 southern San Andreas earthquakes: ground motion sensitivity to rupture speed. Geophysical Research Letters, 35, L22302 (Nov. 2008), doi:10.1029/2008GL035750, 1--5.Google ScholarCross Ref
- Graves, R., Jordan, T. H., Callaghan, S. Deelman, E., Field, E., Juve, G., Kesselman, C., Maechling, P. Mehta, G., Milner, K., Okaya, D., Small, P. and Vahi, K. 2011. CyberShake: A physics-based seismic hazard model for southern california. Pure and Applied Geophysics, vol. 168, no. 3 (Mar. 2011), 367--381.Google Scholar
- Mai, P. M., Imperatori, W. and Olsen, K. B. 2010. Hybrid broadband ground motion simulations: combining long-period deterministic synthetics with high frequency multiple S-to-S back-scattering. Bulletin of the Seismological Society of America, 100, 5A, 2124--2142.Google ScholarCross Ref
- National Research Council 2011. National earthquake resilience: Research, implementation, and outreach. National Academies Press, 198.Google Scholar
- Oak Ridge Leadership Computing Facility 2013. Titan User Guide. {Online}. https://www.olcf.ornl.gov/support/system-user-guides/titan-user-guide.Google Scholar
- Olsen, K. B. 1994. Simulation of three-dimensional wave propagation in the Salt Lake basin. University of Utah, Doctoral dissertation.Google Scholar
- Olsen, K. B., Day, S. M., Minster, J. B., Cui, Y., Chourasia, A., Faerman, M., Moore, R., Maechling, P. and Jordan, T. H. 2006. Strong shaking in Los Angeles expected from southern San Andreas earthquake, Geophysical Research Letters, 33, 7 (Apr. 2006).Google ScholarCross Ref
- Olsen, K. B., Day, S. M., Minster, J. B., Cui, Y., Chourasia, Okaya, D., Maechling, P. and Jordan, T. H. 2008. TeraShake2: spontaneous rupture simulations of mw 7.7 earthquakes on the southern San Andreas fault. Bulletin of the Seismological Society of America, vol. 98, no. 3, (Jun. 2008), 1162--1185.Google Scholar
- Porter, K., Hudnut, K., Perry, S., Reichle, M., Scawthorn, C. and Wein, A. 2011. Forward to the special issue on ShakeOut. Earthquake Spectra, 27, 2, 235--237.Google ScholarCross Ref
- Schmedes, J., Archuleta, R. J. and Lavallée, D. 2010. Correlation of earthquake source parameters inferred from dynamic rupture simulations. Journal of Geophysical Research: Solid Earth, 115, B3.Google Scholar
- Shi, Z. and Day, S. M. 2013. Rupture dynamics and ground motion from 3-D rough-fault simulations. Journal of Geophysical research, 118, 1--20.Google Scholar
- Southern California Earthquake Center. 2013. ShakeOut. {Online}. http://shakeout.org.Google Scholar
- Taborda, R. and Bielak, J. 2013. Ground-motion simulation and validation of the 2008 Chino Hills. Bulletin of the Seismological Society of America, 103, 131--156.Google ScholarCross Ref
- Tape, C., Liu, Q., Maggi, A. and Tromp, J. 2010. Seismic tomography of the southern California crust based on spectral-element and adjoint methods. Geophysical Journal International., 180, 1, 433--462.Google ScholarCross Ref
- University of Illinois NCSA 2013. Blue Waters System Overview. {Online}. https://bluewaters.ncsa.illinois.edu/user-guide.Google Scholar
- Wald, D. J. and Graves R. W. 2001. Resolution analysis of finite fault source inversion using one-and three-dimensional Green's functions: 2. Combining seismic and geodetic data. Journal of Geophysical Research, 106.Google ScholarCross Ref
- Zhao, L., Chen, P. and. Jordan, T. H. 2006. Strain Green's tensors, reciprocity, and their applications to seismic source and structure studies. Bulletin of the Seismological Society of America, 96, 5, 1753--1763.Google ScholarCross Ref
- Zhou, J., Unat, D., Choi, D., Guest, C. an Cui, Y. 2012. Hands-on performance tuning of 3D finite difference earthquake simulation on GPU fermi chipset. In Proc. of Int'l. Conf. on Computational Science (ICCS'12, Omaha, Nebraska, June 4--6, 2012), 9, 976--985.Google ScholarCross Ref
- Zhou, J., Cui, Y., Poyraz, E., Choi, D. and Guest, C. 2013. Multi-GPU implementation of a 3D finite difference time domain earthquake code on heterogeneous supercomputers. In Proc. of Int'l. Conf. on Computational Science (ICCS'13, Barcelona, June 5--7, 2013), 18, 1255--1264.Google Scholar
- Physics-based seismic hazard analysis on petascale heterogeneous supercomputers
Recommendations
Adaptive Optimization for Petascale Heterogeneous CPU/GPU Computing
CLUSTER '10: Proceedings of the 2010 IEEE International Conference on Cluster ComputingIn this paper, we describe our experiment developing an implementation of the Linpack benchmark for TianHe-1, a petascale CPU/GPU supercomputer system, the largest GPU-accelerated system ever attempted before. An adaptive optimization framework is ...
High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers
SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and AnalysisThe omission of nonlinear effects in large-scale 3D ground motion estimation, which are particularly challenging due to memory and scalability issues, can result in costly misguidance for structural design in earthquake-prone regions. We have ...
Optimizing linpack benchmark on GPU-accelerated petascale supercomputer
Special issue on Community Analysis and Information RecommendationIn this paper we present the programming of the Linpack benchmark on TianHe-1 system, the first petascale supercomputer system of China, and the largest GPU-accelerated heterogeneous system ever attempted before. A hybrid programming model consisting of ...
Comments