Topologies for improved InfiniBand latency

ABSTRACT

InfiniBand continues to become more and more important in High Performance Computing world. This talk discusses the impact of DDR (i.e., 20 GigE) switches and HCA's on the creation of low latency, high bandwidth InfiniBand fabrics. When combined with low latency HCA's, such as those from Qlogic, the fabrics discussed can make as much as a 40% reduction in fabric latency, improving the performance of fine grain parallel applications. They also make it possible to create 3 hop low latency fabrics that provide excellent performance that can be used with clusters that have as many as 1009 nodes.There are two approaches to using DDR fabrics to improving latency. The first uses a combination of one, two and three hop fabrics and uses what we call a FasTree topology to create fabrics which can be used to create small clusters (32 to 96 nodes). FasTree's not only have lower latency, but require fewer switch components than fat trees. This does not mean that they have smaller bi-sectional bandwidths, as their links run at twice the speed of an SDR fabric. One of the features of a FasTree, that distinguishes it from other fabrics, is that it does not contain spines. The second fabric, which we call a ThinTree, uses complex single hop spines to link together different fabric sub-domains. Any node in a ThinTree is at most 3 hops away from any other node. There are, however, some compromises required to make it possible to link together up to 1008 nodes without exceeding 3 hops. These compromises result in sub-domains whose intra-domain bandwidth is full CBB while their inter-domain bandwidth typically runs around 40% of CBB. However, because of the 1.8 GB/sec bandwidth of the DDR fabrics, all the connections between any two nodes in a ThinTree fabric are adequate for virtually any HPC application and most others as well. The characteristics of both these topologies are discussed in the talk.

Index Terms

• Computer systems organization

  • Architectures

    • Parallel architectures

      • Interconnection architectures

  • Dependable and fault-tolerant systems and networks

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Hardware

  • Communication hardware, interfaces and storage

    • Buses and high-speed links

• Networks

  • Network performance evaluation