Integrated network interfaces for high-bandwidth TCP/IP

ABSTRACT

This paper proposes new network interface controller (NIC) designs that take advantage of integration with the host CPU to provide increased flexibility for operating system kernel-based performance optimization.We believe that this approach is more likely to meet the needs of current and future high-bandwidth TCP/IP networking on end hosts than the current trend of putting more complexity in the NIC, while avoiding the need to modify applications and protocols. This paper presents two such NICs. The first, the simple integrated NIC (SINIC), is a minimally complex design that moves the responsibility for managing the network FIFOs from the NIC to the kernel. Despite this closer interaction between the kernel and the NIC, SINIC provides performance equivalent to a conventional DMA-based NIC without increasing CPU overhead. The second design, V-SINIC, adds virtual per-packet registers to SINIC, enabling parallel packet processing while maintaining a FIFO model. V-SINIC allows the kernel to decouple examining a packet's header from copying its payload to memory. We exploit this capability to implement a true zero-copy receive optimization in the Linux 2.6 kernel, providing bandwidth improvements of over 50% on unmodified sockets-based receive-intensive benchmarks.

REFERENCES

• 1: Alacritech, Inc. Alacritech / SLIC technology overview. http://www.alacritech.com/html/tech review.html.
• 2: Apache Software Foundation. Apache HTTP server. http://httpd.apache.org.
• 3: Paul Barford , Mark Crovella, Generating representative Web workloads for network and server performance evaluation, Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, p.151-160, June 22-26, 1998, Madison, Wisconsin, United States
• 4: Nathan L. Binkert , Ronald G. Dreslinski , Lisa R. Hsu , Kevin T. Lim , Ali G. Saidi , Steven K. Reinhardt, The M5 Simulator: Modeling Networked Systems, IEEE Micro, v.26 n.4, p.52-60, July 2006 [doi>10.1109/MM.2006.82]
• 5: Nathan L. Binkert , Lisa R. Hsu , Ali G. Saidi , Ronald G. Dreslinski , Andrew L. Schultz , Steven K. Reinhardt, Performance Analysis of System Overheads in TCP/IP Workloads, Proceedings of the 14th International Conference on Parallel Architectures and Compilation Techniques, p.218-230, September 17-21, 2005 [doi>10.1109/PACT.2005.35]
• 6: M. A. Blumrich , C. Dubnicki , E. W. Felten , Kai Li, Protected, user-level DMA for the SHRIMP network interface, Proceedings of the 2nd IEEE Symposium on High-Performance Computer Architecture, p.154, February 03-07, 1996
• 7: Broadcom Corp. BCM5706 product brief, 2004. http://www.broadcom.com/collateral/pb/5706-PB04-R.pdf.
• 8: Broadcom Corporation. BCM1250 product brief, 2003. http://www.broadcom.com/collateral/pb/1250-PB09-R.pdf.
• 9: J. Chase. High Performance TCP/IP Networking, chapter 13, "Software Implementation of TCP". Prentice-Hall, 2003.
• 10: J. Corbet. Linux and TCP offload engines. Linux Weekly News, Aug. 2005. http://lwn.net/Articles/148697.
• 11: W. J. Dally , L. Chao , A. Chien , S. Hassoun , W. Horwat , J. Kaplan , P. Song , B. Totty , S. Wills, Architecture of a message-driven processor, Proceedings of the 14th annual international symposium on Computer architecture, p.189-196, June 02-05, 1987, Pittsburgh, Pennsylvania, United States [doi>10.1145/30350.30372]
• 12: C. Dalton, G. Watson, D. Banks, C. Calamvokis, A. Edwards, and J. Lumley. Afterburner. IEEE Network, 7(4):36--43, July 1993.
• 13: C. Demerjian. Sun's Niagara falls neatly into multithreaded place. The Inquirer, Nov. 2004. http://www.theinquirer.net/?article=19423.
• 14: Wu-chun Feng , Justin (Gus) Hurwitz , Harvey Newman , Sylvain Ravot , R. Les Cottrell , Olivier Martin , Fabrizio Coccetti , Cheng Jin , Xiaoliang (David) Wei , Steven Low, Optimizing 10-Gigabit Ethernet for Networks of Workstations, Clusters, and Grids: A Case Study, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.50, November 15-21, 2003
• 15: Marco Fillo , Stephen W. Keckler , William J. Dally , Nicholas P. Carter , Andrew Chang , Yevgeny Gurevich , Whay S. Lee, The M-Machine multicomputer, Proceedings of the 28th annual international symposium on Microarchitecture, p.146-156, November 29-December 01, 1995, Ann Arbor, Michigan, United States
• 16: A. P. Foong , T. R. Huff , H. H. Hum , J. R. Patwardhan , G. J. Regnier, TCP performance re-visited, Proceedings of the 2003 IEEE International Symposium on Performance Analysis of Systems and Software, p.70-79, March 06-08, 2003
• 17: B. Francis. Enterprises pushing 10GigE to edge. InfoWorld, Dec. 2004. http://www.infoworld.com/article/04/12/06/49NNcisco 1.html.
• 18: D. Freimuth, E. Hu, J. LaVoie, R. Mraz, E. Nahum, P. Pradhan, and J. Tracey. Server network scalability and TCP offload. In Proc. 2005 USENIX Technical Conference, pages 209--222, Apr. 2005.
• 19: A. Gallatin, J. Chase, and K. Yocum. Trapeze/IP: TCP/IP at neargigabit speeds. In Proc. 1999 USENIX Technical Conference, Freenix Track, 1999.
• 20: P. Gelsinger, H.G. Geyer, and J. Rattner. Speeding up the network: A system problem, a platform solution. Technology@Intel Magazine, Mar. 2005. http://www.intel.com/technology/magazine/communications/speeding-network-0305.pdf.
• 21: Dana S. Henry , Christopher F. Joerg, A tightly-coupled processor-network interface, Proceedings of the fifth international conference on Architectural support for programming languages and operating systems, p.111-122, October 12-15, 1992, Boston, Massachusetts, United States
• 22: Hewlett-Packard Company. Netperf: A network performance benchmark. http://www.netperf.org.
• 23: L.R. Hsu, A.G. Saidi, N.L. Binkert, and S.K. Reinhardt. Sampling and stability in TCP/IP workloads. In Proc. First Annual Workshop on Modeling, Benchmarking, and Simulation, pages 68--77, June 2005.
• 24: Ram Huggahalli , Ravi Iyer , Scott Tetrick, Direct Cache Access for High Bandwidth Network I/O, Proceedings of the 32nd Annual International Symposium on Computer Architecture, p.50-59, June 04-08, 2005
• 25: Intel Corp. Intel IXP1200 Network Processor Family - Hardware Reference Manual, Dec. 2001.
• 26: K. Lauritzen, T. Sawicki, T. Stachura, and C.E. Wilson. Intel I/O acceleration technology improves network performance, reliability and efficiently. Technology@Intel magazine, Mar. 2005. http://www.intel.com/technology/magazine/communications/Intel-IOAT-0305.pdf.
• 27: D.S. Miller. Re: {PATCH} TCP Offload (TOE) - Chelsio. E-mail, Aug. 2005. http://lwn.net/Articles/148701.
• 28: J.C. Mogul. TCP offload is a dumb idea whose time has come. In Proc. 9th Workshop on Hot Topics in Operating Systems, May 2003.
• 29: Shubhendu S. Mukherjee , Mark D. Hill, Making Network Interfaces Less Peripheral, Computer, v.31 n.10, p.70-76, October 1998 [doi>10.1109/2.722292 ]
• 30: T. H. Myer , I. E. Sutherland, On the design of display processors, Communications of the ACM, v.11 n.6, p.410-414, June 1968 [doi>10.1145/363347.363368]
• 31: National Semiconductor. DP83820 datasheet, Feb. 2001. http://www.national.com/ds.cgi/DP/DP83820.pdf.
• 32: R. S. Nikhil , G. M. Papadopoulos , Arvind, T: a multithreaded massively parallel architecture, Proceedings of the 19th annual international symposium on Computer architecture, p.156-167, May 19-21, 1992, Queensland, Australia
• 33: M. Ohmacht et al. Blue Gene/L compute chip: Memory and Ethernet subsystem. IBM Journal of Research and Development, 49(2/3):255--264, March/May 2005.
• 34: Greg Regnier , Srihari Makineni , Ramesh Illikkal , Ravi Iyer , Dave Minturn , Ram Huggahalli , Don Newell , Linda Cline , Annie Foong, TCP Onloading for Data Center Servers, Computer, v.37 n.11, p.48-58, November 2004 [doi>10.1109/MC.2004.223]
• 35: A.G. Saidi, N.L. Binkert, L.R. Hsu, and S.K. Reinhardt. Performance validation of network-intensive workloads on a fullsystem simulator. In Proc. 2005 Workshop on Interaction between Operating System and Computer Architecture (IOSCA), pages 33--38, Oct. 2005.
• 36: J. Satran, C. Sapuntzakis, M. Chadalapaka, and E. Zeidner. iscsi. http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-20. pdf, January 2004.
• 37: Piyush Shivam , Jeffrey S. Chase, On the elusive benefits of protocol offload, Proceedings of the ACM SIGCOMM workshop on Network-I/O convergence: experience, lessons, implications, August 25-27, 2003, Karlsruhe, Germany [doi>10.1145/944747.944750]
• 38: Standard Performance Evaluation Corporation. SPECweb99 benchmark. http://www.spec.org/web99.
• 39: Paul Willmann , Hyong-youb Kim , Scott Rixner , Vijay S. Pai, An Efficient Programmable 10 Gigabit Ethernet Network Interface Card, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.96-107, February 12-16, 2005 [doi>10.1109/HPCA.2005.6]

INDEX TERMS

Primary Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices

Additional Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.5 Servers

General Terms:
Design, Experimentation, Performance

Keywords:
TCP/IP performance, interfaces, network, zero-copy