skip to main content
10.1145/1654059.1654128acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

Enabling high-fidelity neutron transport simulations on petascale architectures

Published: 14 November 2009 Publication History

Abstract

The UNIC code is being developed as part of the DOE's Nuclear Energy Advanced Modeling and Simulation (NEAMS) program. UNIC is an unstructured, deterministic neutron transport code that allows a highly detailed description of a nuclear reactor. The primary goal of our simulation efforts is to reduce the uncertainties and biases in reactor design calculations by progressively replacing existing multilevel averaging (homogenization) techniques with more direct solution methods based on first principles. Since the neutron transport equation is seven dimensional (three in space, two in angle, one in energy, and one in time), these simulations are among the most memory and computationally intensive in all of computational science. In order to model the complex physics of a reactor core, billions of spatial elements, hundreds of angles, and thousands of energy groups are necessary, leading to problem sizes with petascale degrees of freedom. Therefore, these calculations exhaust memory resources on current and even next-generation architectures. In this paper, we present UNIC simulation results for two important representative problems in reactor design and analysis---PHENIX and ZPR-6. In each case, UNIC shows good weak scalability on up to 163,840 cores of Blue Gene/P (Argonne) and 122,800 cores of XT5 (Oak Ridge). While our current per processor performance is less than ideal, we demonstrate a clear ability to effectively utilize the leadership computing platforms. Over the coming months, we aim to improve the per processor performance while maintaining the high parallel efficiency by employing better algorithms such as spatial p- and h-multigrid preconditioners, optimized matrix-tensor operations, and weighted partitioning for better load balancing. Combining these additional algorithmic improvements with the availability of larger parallel machines should allow us to realize our long-term goal of explicit geometry coupled multiphysics reactor simulations. In the long run, these high-fidelity simulations will be able to replace expensive mockup experiments and reduce the uncertainty in crucial reactor design and operational parameters.

References

[1]
E. E. Lewis and Jr. W. F. Miller, "Computational Methods of Neutron Transport," John Wiley&Sons, New York, 1984.
[2]
G. Palmiotti, M. A. Smith, C. Rabiti, M. Leclere, D. Kaushik, A. Siegel, B. Smith, E. E. Lewis, "UNÌC: Ultimate Neutronic Investigation Code," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
[3]
Argonne National Laboratory, Argonne Leadership Computing Facility, http://www.alcf.anl.gov.
[4]
Oak Ridge National Laboratory, National Center for Computational Sciences, http://www.nccs.gov.
[5]
C. Rabiti, E. Wolters, M. A. Smith, G. Palmiotti, "Spherical Quadratures for the Discrete Ordinates Method," Trans. Am. Nucl. Soc 96, Boston, MA, June 24--28, 2007.
[6]
J. Lamarsh, "Introduction to Nuclear Engineering," Addison-Wesley Publishing Co, Massachusetts, 1983.
[7]
R. E. Alcouffe, R. S. Baker, J. A. Dahl, S. A. Turner, and Robert Ward, "PARTISN: A Time-Dependent, Parallel Neutral Particle Transport Code System," LA-UR-05-3925, May 2005.
[8]
W. A. Rhoades, R. L. Childs, M. B. Emmett, and S. N. Cramer, "Application of the Three-Dimensional Oak Ridge Transport Code," Proc. Am. Nucl. Soc. Topical Meeting on Reactor Physics and Shielding," pp. 225--238, Chicago, September 17--19, 1984.
[9]
Sjoden, G. and Haghighat, A., "PENTRAN#8482;: Parallel Environment Neutral particle TRANsport in 3-D Cartesian Geometry", Proc. Int. Conf. on Mathematical Methods and Supercomputing for Nuclear Applications, pp. 232--234. Saratoga Springs, NY (1997). Performance of PENTRAN#8482; 3-D Parallel Particle Transport Code.
[10]
G. Palmiotti, E. E. Lewis and C. B. Carrico, "VARIANT: VARIational Anisotropic Nodal Transport for Multidimensional Cartesian and Hexagonal Geometry Calculation," ANL-95/40 Argonne National Laboratory, 1995.
[11]
C. R. E. de Oliveira, A. J. H. Goddard, "EVENT - A Multidimensional Finite Element-Spherical Harmonics Radiation Transport Code," Proceedings of the OECD International Seminar on 3D Deterministic Radiation Transport Codes, Paris, December 01--02, 1996.
[12]
John M. McGhee, Randy M. Roberts, Jim E. Morel, "The DANTE Boltzmann Transport Sove: An Unstructured Mesh, 3-D, Spherical Harmonics Algorithm Compatible with Parallel Computer Architectures," Jnt. Int. Conf. on Math. Meth. and super. For Nuc. Apps., Saratoga Springs, New York, Oct. 5--10, 1997.
[13]
T. A. Wareing, J. M. McGhee, J. E. Morel, and S. D. Pautz, "Discontinuous Finite Element Sn Methods on 3-D Unstructured Meshes," Nuclear Science and Engineering, 138:1--13, 2001.
[14]
A. Siegel, D. Kaushik, P. Fischer, G. Palmiotti, M. A. Smith, C. Rabiti, J. Ragusa, "Software Design of SHARP," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
[15]
Rabiti, M. A. Smith, G. Palmiotti, "A Three-dimensional Method of Characteristics on Unstructured Tetrahedral Meshes," Trans. Am. Nucl. Soc. 96, 470, 2007.
[16]
M. A. Smith, G. Palmiotti, C. Rabiti, D. Kaushik, A. Siegel, B. Smith, E. E. Lewis, "PNFE Component of the UNÌC Code," Joint International Topical Meeting on Mathematics&Computation and Supercomputing in Nuclear Applications, Monterey, California, April 15--19, 2007.
[17]
S. Balay, K. R. Buschelman, W. D. Gropp, D. K. Kaushik, M. G. Knepley, L. C. McInnes, and B. F. Smith, "PETSc Home Page," http://www.mcs.anl.gov/petsc.
[18]
M. A. Smith, G. Palmiotti, et al., "Benchmark on Deterministic Transport Calculations Without Spatial Homogenization (MOX Fuel Assembly 3-D Extension Case)," OECD/NEA document, NEA/NSC/DOC(2005)16, October 2005.
[19]
Personal Communication, Frederic Varaine, CEA, "IAEA CRP on PHENIX End of Life Tests; Control Rod Withdrawal," 2009.
[20]
Lawrence Livermore National Laboratory, "Visit: A Distributed, Parallel, Visualization Tool," https://wci.llnl.gov/codes/visit/home.html.
[21]
Sandia National Laboratory, "CUBIT: Geometry and Mesh Generation Toolkit," http://cubit.sandia.gov/.
[22]
Amine Abou-Rjeili and George Karypis, "Multilevel Algorithms for Partitioning Power-Law Graphs," IEEE International Parallel&Distributed Processing Symposium (IPDPS), 2006.
[23]
NEA Nuclear Science Committee, "International Handbook of Evaluated Criticality Safety Benchmark Experiments," NEA/NSC/DOC(95)03, September 2007.
[24]
W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith, "Toward Realistic Performance Bounds for Implicit CFD Codes," In D. Keyes, A. Ecer, J. Periaux, N. Satofuka, and P. Fox, editors, Proceedings of Parallel CFD'99, pages 233--240. Elsevier, 1999.
[25]
D. Kaushik, W. Gropp, M. Minkoff, B. Smith, "Improving the Performance of Tensor Matrix Vector Multiplication in Cumulative Reaction Probability Based Quantum Chemistry Codes," Proceedings of the 15th International Conference on High Performance Computing (HiPC 2008), Bangalore, India, Springer LNCS5374:120--130, December, 2008.

Cited By

View all
  • (2018)Hierarchical Krylov and nested Krylov methods for extreme-scale computingParallel Computing10.1016/j.parco.2013.10.00140:1(17-31)Online publication date: 31-Dec-2018
  • (2015)3D Cartesian Transport Sweep for Massively Parallel Architectures with PaRSECProceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium10.1109/IPDPS.2015.75(581-590)Online publication date: 25-May-2015
  • (2013)Multiphysics simulationsThe International Journal of High Performance Computing Applications10.1177/109434201246818127:1(4-83)Online publication date: 7-Feb-2013
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SC '09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
November 2009
778 pages
ISBN:9781605587448
DOI:10.1145/1654059
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. discrete ordinates
  2. memory bandwidth
  3. neutron transport
  4. nuclear reactors
  5. parallel scalability

Qualifiers

  • Research-article

Funding Sources

Conference

SC '09
Sponsor:

Acceptance Rates

SC '09 Paper Acceptance Rate 59 of 261 submissions, 23%;
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)0
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2018)Hierarchical Krylov and nested Krylov methods for extreme-scale computingParallel Computing10.1016/j.parco.2013.10.00140:1(17-31)Online publication date: 31-Dec-2018
  • (2015)3D Cartesian Transport Sweep for Massively Parallel Architectures with PaRSECProceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium10.1109/IPDPS.2015.75(581-590)Online publication date: 25-May-2015
  • (2013)Multiphysics simulationsThe International Journal of High Performance Computing Applications10.1177/109434201246818127:1(4-83)Online publication date: 7-Feb-2013
  • (2013)A domain decomposition approach to implementing fault slip in finite‐element models of quasi‐static and dynamic crustal deformationJournal of Geophysical Research: Solid Earth10.1002/jgrb.50217118:6(3059-3079)Online publication date: 10-Jun-2013
  • (2012)Improving High-Performance Sparse Libraries Using Compiler-Assisted SpecializationProceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum10.1109/IPDPSW.2012.63(487-496)Online publication date: 21-May-2012
  • (2011)Hybrid programming model for implicit PDE simulations on multicore architecturesProceedings of the 7th international conference on OpenMP in the Petascale era10.5555/2023025.2023028(12-21)Online publication date: 13-Jun-2011
  • (2011)Hybrid Programming Model for Implicit PDE Simulations on Multicore ArchitecturesOpenMP in the Petascale Era10.1007/978-3-642-21487-5_2(12-21)Online publication date: 2011

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media