skip to main content
10.1145/3126908.3126967acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article
Public Access

Predicting the performance impact of different fat-tree configurations

Published: 12 November 2017 Publication History

Abstract

The fat-tree topology is one of the most commonly used network topologies in HPC systems. Vendors support several options that can be configured when deploying fat-tree networks on production systems, such as link bandwidth, number of rails, number of planes, and tapering. This paper showcases the use of simulations to compare the impact of these design options on representative production HPC applications, libraries, and multi-job workloads. We present advances in the TraceR-CODES simulation framework that enable this analysis and evaluate its prediction accuracy against experiments on a production fat-tree network. In order to understand the impact of different network configurations on various anticipated scenarios, we study workloads with different communication patterns, computation-to-communication ratios, and scaling characteristics. Using multi-job workloads, we also study the impact of inter-job interference on performance and compare the cost-performance tradeoffs.

References

[1]
2015. Performance Scaled Messaging 2 (Intel). https://www.intel.com/content/dam/support/us/en/documents/network/omni-adptr/sb/Intel_PSM2_PG_H76473_v1_0.pdf. (2015).
[2]
2015. Score-P User Manual. https://silc.zih.tu-dresden.de/scorep-current/pdf/scorep.pdf. (2015).
[3]
2017. Catalyst (LLNL). https://hpc.llnl.gov/hardware/platforms/catalyst. (2017).
[4]
2017. MFEM: Modular finite element methods. mfem.org. (2017).
[5]
2017. MULARD. (2017). https://codesign.llnl.gov/mulard.php.
[6]
2017. Open Trace Format 2. (2017). https://silc.zih.tu-dresden.de/otf2-current/index.html.
[7]
2017. Quartz (LLNL). https://hpc.llnl.gov/hardware/platforms/Quartz. (2017).
[8]
2017. The Sierra Advanced Technology System. http://computation.llnl.gov/computers/sierra-advanced-technology-system. (2017).
[9]
2017. Stampede (TACC). https://www.tacc.utexas.edu/stampede/. (2017).
[10]
2017. Summit (OLCF). http://www.olcf.ornl.gov/summit. (2017).
[11]
Bilge Acun, Nikhil Jain, Abhinav Bhatele, Misbah Mubarak, Christopher D. Carothers, and Laxmikant V. Kale. 2015. Preliminary Evaluation of a Parallel Trace Replay Tool for HPC Network Simulations. In Proceedings of the 3rd Workshop on Parallel and Distributed Agent-Based Simulations (PADABS '15). LLNL-CONF-667225.
[12]
Albert Alexandrov, Mihai F. Ionescu, Klaus E. Schauser, and Chris Scheiman. 1995. LogGP: incorporating long messages into the LogP modelfi?!one step closer towards a realistic model for parallel computation. In Proceedings of the seventh annual ACM symposium on Parallel algorithms and architectures (SPAA '95). ACM, New York, NY, USA, 95--105.
[13]
A. Arsenlis, W. Cai, M. Tang, M. Rhee, T. Oppelstrup, G. Hommes, T. G. Pierce, and V. V. Bulatov. 2007. Enabling strain hardening simulations with dislocation dynamics. Modelling and Simulation in Materials Science and Engineering 15, 6 (2007).
[14]
David W. Bauer Jr., Christopher D. Carothers, and Akintayo Holder. 2009. Scalable Time Warp on Blue Gene Supercomputers. In Proceedings of the 2009 ACM/IEEE/SCS 23rd Workshop on Principles of Advanced and Distributed Simulation (PADS '09). IEEE Computer Society, Washington, DC, USA.
[15]
Claude Bernard, Tom Burch, Tomas A. DeGrand, Carleton DeTar, Steven Gottlieb, Urs M. Heller, James E. Hetrick, Kostas Orginos, Bob Sugar, and Doug Toussaint. 2000. Scaling tests of the improved Kogut-Susskind quark action. Physical Review D 61 (2000).
[16]
Patrick S. Brantley, Shawn A. Dawson, Michael Scott McKinley, Matthew J. O'Brien, David E. Stevens, Bret R. Beck, Eric D. Jurgenson, Chris A. Ebbers, and James M. Hall. 2013. Recent Advances in the Mercury Monte Carlo Particle Transport Code. In International Conference on Mathematics and Computational Methods Applied to Nuclear Science & Engineering (M&C'13). Sun Valley, ID.
[17]
Ron Brightwell and Keith Underwood. 2003. Evaluation of an eager protocol optimization for MPI. In European Parallel Virtual Machine/Message Passing Interface Usersfi Group Meeting. Springer, 327--334.
[18]
Henri Casanova, Arnaud Giersch, Arnaud Legrand, Martin Quinson, and Frédéric Suter. 2014. Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms. J. Parallel and Distrib. Comput. 74, 10 (June 2014), 2899--2917.
[19]
Dong Chen, N.A. Eisley, P. Heidelberger, R.M. Senger, Y. Sugawara, S. Kumar, V. Salapura, D.L. Satterfield, B. Steinmacher-Burow, and J.J. Parker. 2011. The IBM Blue Gene/Q interconnection network and message unit. In High Performance Computing, Networking, Storage and Analysis (SC), 2011 International Conference for. 1--10.
[20]
Salvador Coll, Eitan Frachtenberg, Fabrizio Petrini, Adolfy Hoisie, and Leonid Gurvits. 2003. Using multirail networks in high-performance clusters. Concurrency and Computation: Practice and Experience 15, 7-8 (2003), 625--651.
[21]
Greg Faanes, Abdulla Bataineh, Duncan Roweth, Tom Court, Edwin Froese, Bob Alverson, Tim Johnson, Joe Kopnick, Mike Higgins, and James Reinhard. 2012. Cray Cascade: A Scalable HPC System Based on a Dragonfly Network. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC '12). IEEE Computer Society Press, Los Alamitos, CA, USA.
[22]
R.D. Falgout, J.E. Jones, and U.M. Yang. 2006. The Design and Implementation of hypre, a Library of Parallel High Performance Preconditioners. In Numerical Solution of Partial Differential Equations on Parallel Computers, A.M. Bruaset and A. Tveito (Eds.). Vol. 51. Springer-Verlag, 267--294.
[23]
F. Gygi, E. W. Draeger, B. R. De Supinski, R. K. Yates, F. Francheti, S. Kral, J. Lorenz, C. W. Ueberhuber, J. A. Gunnels, and J. C. Sexton. 2005. Large-Scale First-Principles Molecular Dynamics Simulations on the Blue Gene/L Platform using the Qbox Code. In Proceedings of Supercomputing 2005 4 (2005), 24. Conference on High Performance Networking and Computing, Gordon Bell Prize finalist.
[24]
John P. Hayes, Trevor N. Mudge, and Quentin F. Stout. 1986. Architecture of a Hypercube Supercomputer. In ICPP. 653--660.
[25]
Chao Huang, Gengbin Zheng, Sameer Kumar, and Laxmikant V. Kalé. 2006. Performance Evaluation of Adaptive MPI. In Proceedings of ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming 2006.
[26]
Nikhil Jain, Abhinav Bhatele, Xiang Ni, Nicholas J. Wright, and Laxmikant V. Kale. 2014. Maximizing Throughput on a Dragonfly Network. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '14). IEEE Computer Society. LLNL-CONF-653557.
[27]
Nikhil Jain, Abhinav Bhatele, Samuel T. White, Todd Gamblin, and Laxmikant V. Kale. 2016. Evaluating HPC Networks via Simulation of Parallel Workloads. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '16). IEEE Computer Society. LLNL-CONF-690662.
[28]
Nan Jiang, Daniel U. Becker, George Michelogiannakis, James Balfour, Brian Towles, John Kim, and William J. Dally. 2013. A Detailed and Flexible Cycle-Accurate Network-on-Chip Simulator. In IEEE International Symposium on Performance Analysis of Systems and Software.
[29]
John Kim, Wiliam J. Dally, Steve Scott, and Dennis Abts. 2008. Technology- Driven, Highly-Scalable Dragonfly Topology. SIGARCH Comput. Archit. News 36 (June 2008), 77--88. Issue 3.
[30]
C.E. Leiserson. 1985. Fat-trees: Universal Networks for Hardware-Efficient Supercomputing. IEEE Transactions on Computers 34, 10 (October 1985).
[31]
Charles E. Leiserson. 1985. Fat-trees: Universal Networks for Hardware-efficient Supercomputing. IEEE Trans. Comput. 34, 10 (Oct. 1985), 892--901.
[32]
Edgar A. Leon, Ian Karlin, Abhinav Bhatele, Steven H. Langer, Chris Chambreau, Louis H. Howell, Trent D'Hooge, and Matthew L. Leininger. 2016. Characterizing Parallel Scientific Applications on Commodity Clusters: An Empirical Study of a Tapered Fat-tree. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '16). IEEE Computer Society, Article 78, 12 pages. http://dl.acm.org/citation.cfm?id=3014904.3015009 LLNL-CONF-681011.
[33]
M.Blumrich, D.Chen, P.Coteus, A.Gara, M.Giampapa, P.Heidelberger, S.Singh, B.Steinmacher-Burow, T.Takken, and P.Vranas. 2003. Design and Analysis of the Blue Gene/L Torus Interconnection Network. IBM Research Report (December 2003).
[34]
George Michelogiannakis, Khalid Ibrahim, Jeremiah Shalf, John anWilke, Samuel Knight, and Joseph Kenny. 2017. APHiD: Hierarchical Task Placement to Enable a Tapered Fat Tree Topology for Lower Power and Cost in HPC Networks. CCGrid 2017 (to appear) (2017).
[35]
Misbah Mubarak, Christopher D. Carothers, Robert B. Ross, and Philip Carns. 2016. Enabling Parallel Simulation of Large-Scale HPC Network Systems. IEEE Trans. Parallel Distrib. Syst. (2016).
[36]
Mohamed Ould-Khaoua and Hamid Sarbazi-Azad. 2001. An Analytical Model of Adaptive Wormhole Routing in Hypercubes in the Presence of Hot Spot Traffic. IEEE Transactions on Parallel and Distributed Systems 12, 3 (2001), 283--292.
[37]
S. Shende and A. D. Malony. 2005. The TAU Parallel Performance System. International Journal of High Performance Computing Applications, ACTS Collection Special Issue (2005).
[38]
Kyle L. Spafford and Jeffrey S. Vetter. 2012. Aspen: A Domain Specific Language for Performance Modeling. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC '12). IEEE Computer Society Press, Los Alamitos, CA, USA, Article 84, 11 pages. http://dl.acm.org/citation.cfm?id=2388996.2389110
[39]
C. H. Still, R. L. Berger, A. B. Langdon, D. E. Hinkel, L. J. Suter, and E. A. Williams. 2000. Filamentation and forward Brillouin scatter of entire smoothed and aberrated laser beams. Physics of Plasmas 7, 5 (2000), 2023--2032.
[40]
Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Optimization of Collective Communication Operations in MPICH. International Journal of High Performance Computing Applications 19, 1 (2005), 49--66.
[41]
K.D. Underwood, M. Levenhagen, and A. Rodrigues. 2007. Simulating Red Storm: Challenges and Successes in Building a System Simulation. In IEEE International Parallel and Distributed Processing Symposium (IPDPS '07).
[42]
Noah Wolfe, Misbah Mubarak, Nikhil Jain, Jens Domke, Abhinav Bhatele, Christopher Carothers, and Rob Ross. 2017. Methods for Effective Utilization of Multi-Rail Fat-Tree Interconnects (CCGrid 2017 (to appear)).
[43]
Xu Yang, John Jenkins, Misbah Mubarak, Robert B. Ross, and Zhiling Lan. 2016. Watch Out for the Bully! Job Interference Study on Dragonfly Network. In Supercomputing.

Cited By

View all
  • (2024)SDN dynamic flow scheduling algorithm based on discrete particle swarmScientific Insights and Discoveries Review10.59782/sidr.v1i1.361:1Online publication date: 26-Sep-2024
  • (2024)Trade-off topology design for hierarchical network based on job characteristicsCCF Transactions on High Performance Computing10.1007/s42514-024-00193-zOnline publication date: 21-May-2024
  • (2024)Analysis and prediction of performance variability in large-scale computing systemsThe Journal of Supercomputing10.1007/s11227-024-06040-wOnline publication date: 28-Mar-2024
  • Show More Cited By

Index Terms

  1. Predicting the performance impact of different fat-tree configurations

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
        November 2017
        801 pages
        ISBN:9781450351140
        DOI:10.1145/3126908
        • General Chair:
        • Bernd Mohr,
        • Program Chair:
        • Padma Raghavan
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        In-Cooperation

        • IEEE CS

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 12 November 2017

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. fat-tree topology
        2. network simulation
        3. performance prediction
        4. procurement

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        SC '17
        Sponsor:

        Acceptance Rates

        SC '17 Paper Acceptance Rate 61 of 327 submissions, 19%;
        Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

        Upcoming Conference

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)229
        • Downloads (Last 6 weeks)20
        Reflects downloads up to 19 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)SDN dynamic flow scheduling algorithm based on discrete particle swarmScientific Insights and Discoveries Review10.59782/sidr.v1i1.361:1Online publication date: 26-Sep-2024
        • (2024)Trade-off topology design for hierarchical network based on job characteristicsCCF Transactions on High Performance Computing10.1007/s42514-024-00193-zOnline publication date: 21-May-2024
        • (2024)Analysis and prediction of performance variability in large-scale computing systemsThe Journal of Supercomputing10.1007/s11227-024-06040-wOnline publication date: 28-Mar-2024
        • (2023)A Weighted Optimal Scheduling Scheme for Congestion Control in Cloud Data Center NetworksIEEE Transactions on Services Computing10.1109/TSC.2023.323952416:4(2402-2410)Online publication date: 1-Jul-2023
        • (2023)Understanding Node Allocation on Leadership-Class Supercomputers with Graph Analytics2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys)10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00113(780-787)Online publication date: 17-Dec-2023
        • (2022)Effective TCP Flow Management Based on Hierarchical Feedback Learning in Complex Data Center NetworkSensors10.3390/s2202061122:2(611)Online publication date: 13-Jan-2022
        • (2022)Mixed-Flow Load-Balanced Scheduling for Software-Defined Networks in Intelligent Video Surveillance Cloud Data CenterApplied Sciences10.3390/app1213647512:13(6475)Online publication date: 26-Jun-2022
        • (2022)Understanding Node Connection Modes in Multi-Rail Fat-treeJournal of Parallel and Distributed Computing10.1016/j.jpdc.2022.04.019Online publication date: May-2022
        • (2022)The Case for Disjoint Job Mapping on High-Radix Networked Parallel ComputersAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-95388-1_9(123-143)Online publication date: 23-Feb-2022
        • (2021)Exploration of Congestion Control Techniques on Dragonfly-class HPC Networks Through Simulation2021 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS)10.1109/PMBS54543.2021.00010(40-50)Online publication date: Nov-2021
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media