On using connection-oriented vs. connection-less transport for performance and scalability of collective and one-sided operations: trade-offs and impact

ABSTRACT

Communication subsystem plays a pivotal role in achieving scalable performance in clusters. The communication semantics employed are dictated by the programming model used by the application such as MPI, UPC, etc. Out of the gamut of communication primitives, collective and one-sided operations are especially significant and have to be designed harnessing the capabilities and features exposed by the underlying networks. In some cases, there is a direct match between the semantics of the operations and the underlying network primitives. InfiniBand provides two transport modes: (i)Connection-oriented Reliable connection (RC) supporting Memory and Channel semantics and (ii) Connection-less Unreliable Datagram (UD) supporting Channel semantics. Achieving good performance and scalability needs careful analysis and design of communication primitives based on these options.

In this paper, we evaluate the scalability and performance trade-offs between RC and UD transport modes. We study the semantic advantages of mapping collective and one-sided operations on to memory and channel semantics of InfiniBand(IBA). We take AlltoAll as a case study to demonstrate the benefits of RDMA over Send/Recv and to show the performance/memory trade-offs over IB transports. Our experimental results show that UD-based AlltoAll performs 38% better than Bruck's algorithm for short messages and up to two times better than the direct AlltoAll over RC. Since InfiniBand does not provide RDMA over UD in hardware, we emulate the same in our study. Our results show a performance dip of up to a factor of three for emulated RDMA Read latency as compared to RC, highlighting the need for hardware based RDMA operations over UD.

REFERENCES

• 1: Rajeev Thakur and William Gropp Improving the Performance of Collective Operations in MPICH In Euro PVM/MPI Conference, 2003.
• 2: Sayantan Sur , Hyun-Wook Jin , Dhabaleswar K. Panda, Efficient and Scalable All-to-All Personalized Exchange for InfiniBand-Based Clusters, Proceedings of the 2004 International Conference on Parallel Processing (ICPP'04), p.275-282, August 15-18, 2004 [doi>10.1109/ICPP.2004.32]
• 3: A. R. Mamidala , Jiuxing Liu , D. K. Panda, Efficient Barrier and Allreduce on Infiniband clusters using multicast and adaptive algorithms, Proceedings of the 2004 IEEE International Conference on Cluster Computing, p.135-144, September 20-23, 2004
• 4: Network-Based Computing Laboratory MVAPICH: MPI for InfiniBand http://nowlab.cse.ohio-state.edu/projects/mpi-iba/index.html
• 5: Abhinav Vishnu and Gopal K. Santhanaraman and Wei Huang and Hyun-Wook Jin and Dhabaleswar K. Panda Supporting MPI-2 One Sided Communication on Multi-Rail InfiniBand Clusters: Design Challenges and Performance Benefits In International Conference on High Performance Computing, HiPC 2005.
• 6: UC Berkeley/LBNL Berkeley UPC - Unified Parallel C, http://upc.lbl.gov/
• 7: Christian Bell and Dan Bonachea and Rajesh Nishtala and Katherine Yelick Optimizing Bandwidth Limited Problems Using One-Sided Communication and Overlap Support In IEEE International Parallel & Distributed Processing Symposium, IPDPS 2006.
• 8: Rajeev Sivaram , Rama K. Govindaraju , Peter Hochschild , Robert Blackmore , Piyush Chaudhary, Breaking the Connection: RDMA Deconstructed, Proceedings of the 13th Symposium on High Performance Interconnects, p.36-42, August 17-19, 2005 [doi>10.1109/CONECT.2005.9]
• 9: InfiniBand Trade Association InfiniBand Architecture Specification, Release 1.1 November,2002, http://www.infinibandta.org
• 10: Jehoshua Bruck , Ching-Tien Ho , Eli Upfal , Shlomo Kipnis , Derrick Weathersby, Efficient Algorithms for All-to-All Communications in Multiport Message-Passing Systems, IEEE Transactions on Parallel and Distributed Systems, v.8 n.11, p.1143-1156, November 1997 [doi>10.1109/71.642949 ]
• 11: William Gropp and Ewing Lusk and Nathan Doss and Anthony Skjellum A High-Performance, Portable Implementation of the MPI, Message Passing Interface Standard Argonne National Laboratory and Mississippi State University, 1996.
• 12: Rajeev Thakur, Rolf Rabenseifner, and William Gropp. Optimization of Collective Communication Operations in MPICH. In Int'l Journal of High Performance Computing Applications, (19)1:49--66, Spring 2005.
• 13: Weihang Jiang, Jiuxing Liu, Hyun-Wook Jin, Dhabaleswar K. Panda, Darius Buntinas, Rajeev Thakur, and William Gropp. Efficient Implementation of MPI-2 Passive One-Sided Communication on InfiniBand Clusters. In Proceedings of the 11th European PVM/MPI Users' Group Meeting (Euro PVM/MPI 2004), Recent Advances in Parallel Virtual Machine and Message Passing Interface, Lecture Notes in Computer Science, LNCS 3241, Springer September 2004, pp. 68--76.
• 14: Rajeev Thakur, William Gropp, and Brian Toonen. Minimizing Synchronization Overhead in the Implementation of MPI One-Sided Communication. In Proceedings of the 11th European PVM/MPI Users' Group Meeting (Euro PVM/MPI 2004), Recent Advances in Parallel Virtual Machine and Message Passing Interface, Lecture Notes in Computer Science, LNCS 3241, Springer, September 2004, pp. 57--67.
• 15: Weihang Jiang , Jiuxing Liu , Hyun-Wook Jin , D. K. Panda , W. Gropp , R. Thakur, High performance MPI-2 one-sided communication over InfiniBand, Proceedings of the 2004 IEEE International Symposium on Cluster Computing and the Grid, p.531-538, April 19-22, 2004
• 16: Amith R. Mamidala, Abhinav Vishnu and Dhabaleswar K. Panda. Efficient Shared Memory and RDMA based design for MPI_Allgather over InfiniBand. In Proceedings of EuroPVM/MPI, September 2006.
• 17: Wei-Yu Chen , Costin Iancu , Katherine Yelick, Communication Optimizations for Fine-Grained UPC Applications, Proceedings of the 14th International Conference on Parallel Architectures and Compilation Techniques, p.267-278, September 17-21, 2005 [doi>10.1109/PACT.2005.13]
• 18: Costin Iancu , Parry Husbands , Paul Hargrove, HUNTing the Overlap, Proceedings of the 14th International Conference on Parallel Architectures and Compilation Techniques, p.279-290, September 17-21, 2005 [doi>10.1109/PACT.2005.25]
• 19: Parry Husbands , Costin Iancu , Katherine Yelick, A performance analysis of the Berkeley UPC compiler, Proceedings of the 17th annual international conference on Supercomputing, June 23-26, 2003, San Francisco, CA, USA [doi>10.1145/782814.782825]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.2 Network Protocols
          Subjects: Protocol architecture (OSI model)

General Terms:
Design, Measurement, Performance

Keywords:
RDMA, reliable connection, send/recv, unreliable datagram