High performance Monte Carlo simulation of ising model on TPU clusters

ABSTRACT

Large-scale deep learning benefits from an emerging class of AI accelerators. Some of these accelerators' designs are general enough for compute-intensive applications beyond AI and Cloud TPU is one such example. In this paper, we demonstrate a novel approach using TensorFlow on Cloud TPU to simulate the two-dimensional Ising Model. TensorFlow and Cloud TPU framework enable the simple and readable code to express the complicated distributed algorithm without compromising the performance. Our code implementation fits into a small Jupyter Notebook and fully utilizes Cloud TPU's efficient matrix operation and dedicated high speed inter-chip connection. The performance is highly competitive: it outperforms the best published benchmarks to our knowledge by 60% in single-core and 250% in multi-core with good linear scaling. When compared to Tesla V100 GPU, the single-core performance maintains a ~10% gain. We also demonstrate that using low precision arithmetic---bfloat16---does not compromise the correctness of the simulation results.

References

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pages 265--283, 2016.
2. Didier Barradas-Bautista, Matias Alvarado-Mentado, Mark Agostino, and Germinal Cocho. Cancer growth and metastasis as a metaphor of go gaming: An ising model approach. PloS one, 13(5):e0195654, 2018.
3. Kurt Binder. Finite size scaling analysis of ising model block distribution functions. Zeitschrift fÃijr Physik B Condensed Matter, 43(2):119--140, 1981.
4. Kurt Binder, Dieter Heermann, Lyle Roelofs, A. John Mallinckrodt, and Susan McKay. Monte carlo simulation in statistical physics. Computers in Physics, 7(2):156--157, 1993.
5. Kurt Binder and Erik Luijten. Monte carlo tests of renormalization-group predictions for critical phenomena in ising models. Physics Reports, 344(4--6):179--253, 2001.
6. Benjamin Block, Peter Virnau, and Tobias Preis. Multi-gpu accelerated multi-spin monte carlo simulations of the 2d ising model. Computer Physics Communications, 181(9):1549--1556, 2010.
7. Google Cloud. Choosing between a single cloud tpu device and a cloud tpu pod (alpha). https://cloud.google.com/tpu/docs/deciding-pod-versus-tpu, 2019.
8. Google Cloud. Performance guide. https://cloud.google.com/tpu/docs/performance-guide, 2019.
9. Google Cloud. System architecture. https://cloud.google.com/tpu/docs/system-architecture, 2019.
10. Google Cloud. Using bfloat16 with tensorflow models. https://cloud.google.com/tpu/docs/bfloat16, 2019.
11. Chase Cook, Hengyang Zhao, Takashi Sato, Masayuki Hiromoto, and Sheldon X.-D. Tan. Gpu based parallel ising computing for combinatorial optimization problems in vlsi physical design. arXiv preprint arXiv:1807.10750, 2018.
12. Nvidia NVLink Fabric. Nvlink fabric: Advancing multi-gpu processing. https://www.nvidia.com/en-us/data-center/nvlink/, 2019.
13. Ernst Ising. Beitrag zur theorie des ferromagnetismus. Zeitschrift fÃijr Physik, 31(1):253--258, 1925.
14. Norm Jouppi. Quantifying the performance of the tpu, our first machine learning chip. Google Cloud: https://cloud.google.com/blog/products/gcp/quantifying-the-performance-of-the-tpu-our-first-machine-learning-chip, 2017.
15. Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, and Al Borchers. In-datacenter performance analysis of a tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on, pages 1--12. IEEE, 2017.
16. Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Landon, Cliff Young, and David Patterson. A Domain-Specific Supercomputer for Training Deep Neural Networks. Submitted to Communications of the ACM.
17. Peter Müller. A generic approach to posterior integration and Gibbs sampling. Purdue University, Department of Statistics, 1991.
18. Lars Onsager. Crystal statistics.i. a two-dimensional model with an order-disorder transition. Physical Review, 65(3--4):117, 1944.
19. Francisco Ortega-Zamorano, Marcelo A. Montemurro, Sergio Alejandro Cannas, José M. Jerez, and Leonardo Franco. Fpga hardware acceleration of monte carlo simulations for the ising model. IEEE Trans. Parallel Distrib. Syst., 27(9):2618--2627, 2016.
20. Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 2017.
21. Tobias Preis, Wolfgang Paul, and Johannes J. Schneider. Fluctuation patterns in high-frequency financial asset returns. EPL (Europhysics Letters), 82(6):68005, 2008.
22. Tobias Preis, Peter Virnau, Wolfgang Paul, and Johannes J. Schneider. Gpu accelerated monte carlo simulation of the 2d and 3d ising model. Journal of Computational Physics, 228(12):4468--4477, 2009.
23. Francisco Prieto-Castrillo, Amin Shokri Gazafroudi, Javier Prieto, and Juan Manuel Corchado. An ising spin-based model to explore efficient flexibility in distributed power systems. Complexity 2018, 2018.
24. Alan M. Ferrenberg, Jiahao Xu, and David P. Landau Pushing the Limits of Monte Carlo Simulations for the 3d Ising Model. arXiv preprint arXiv:1806.03558, 2018.

Index Terms

• Applied computing

  • Physical sciences and engineering

    • Mathematics and statistics

    • Physics

• Computing methodologies

  • Modeling and simulation

    • Simulation types and techniques

      • Distributed simulation

      • Massively parallel and high-performance simulations