Bullet trains: a study of NIC burst behavior at microsecond timescales

ABSTRACT

While numerous studies have examined the macro-level behavior of traffic in data center networks---overall flow sizes, destination variability, and TCP burstiness---little information is available on the behavior of data center traffic at packet-level timescales. Whereas one might assume that flows from different applications fairly share available link bandwidth, and that packets within a single flow are uniformly paced, the reality is more complex. To meet increasingly high link rates of 10 Gbps and beyond, batching is typically introduced across the network stack---at the application, middleware, OS, transport, and NIC layers. This batching results in short-term packet bursts, which have implications for the design and performance requirements of packet processing devices along the path, including middleboxes, SDN-enabled switches, and virtual machine hypervisors.

In this paper, we study the burst behavior of traffic emanating from a 10-Gbps end host across a variety of data center applications. We find that at 10--100 microsecond timescales, the traffic exhibits large bursts (i.e., 10s of packets in length). We further find that this level of burstiness is largely outside of application control, and independent of the high-level behavior of applications.

References

1. 100Gb/s Ethernet Task Force. http://www.ieee802.org/3/ba/.
2. A. Aggarwal, S. Savage, and T. E. Anderson. Understanding the Performance of TCP Pacing. In Proc. INFOCOM, 2000.
3. M. Alizadeh, A. Kabbani, T. Edsall, B. Prabhakar, A. Vahdat, and M. Yasuda. Less is More: Trading a little Bandwidth for Ultra-Low Latency in the Data Center. In Proc. NSDI, 2012.
4. M. Allman and E. Blanton. Notes on Burst Mitigation for Transport Protocols. SIGCOMM Comput. Commun. Rev., 35(2):53--60, Apr. 2005.
5. Apache Software Foundation. HDFS Architecture Guide. http://hadoop.apache.org/docs/hdfs/current/hdfs_design.html.
6. T. Benson, A. Akella, and D. A. Maltz. Network Traffic Characteristics of Data Centers in the Wild. In Proc. IMC, 2010.
7. E. Blanton and M. Allman. On the Impact of Bursting on TCP Performance. In Proc. PAM, 2005.
8. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In Proc. OSDI, 2004.
9. N. Farrington, G. Porter, S. Radhakrishnan, H. H. Bazzaz, V. Subramanya, Y. Fainman, G. Papen, and A. Vahdat. Helios: A Hybrid Electrical/Optical Switch Architecture for Modular Data Centers. In Proc. ACM SIGCOMM, Aug. 2010.
10. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In Proc. SOSP, 2003.
11. M. Ghobadi, Y. Cheng, A. Jain, and M. Mathis. Trickle: Rate Limiting YouTube Video Streaming. In Proc. ATC, 2012.
12. R. Jain and S. Routhier. Packet Trains--Measurements and a New Model for Computer Network Traffic. IEEE J.Sel. A. Commun., 4(6):986--995, Sept. 1986.
13. H. Jiang and C. Dovrolis. Source-level IP Packet Bursts: Causes and Effects. In Proc. IMC, 2003.
14. H. Jiang and C. Dovrolis. Why is the Internet Traffic Bursty in Short Time Scales? In Proc. SIGMETRICS, 2005.
15. M. Moshref, M. Yu, A. Sharma, and R. Govindan. Scalable rule management for data centers. In Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation, nsdi'13, pages 157--170, Berkeley, CA, USA, 2013. USENIX Association.
16. R. N. Mysore, G. Porter, Subramanya, and A. Vahdat. FasTrak: Enabling Express Lanes in Multi-Tenant Data Centers. In Proc. CoNEXT, 2013.
17. ONF. Software-Defined Networking: The New Norm for Networks. https://www.opennetworking.org/.
18. Open vSwitch: An Open Virtual Switch. http://openvswitch.org/.
19. G. Porter, R. Strong, N. Farrington, A. Forencich, P.-C. Sun, T. Rosing, Y. Fainman, G. Papen, and A. Vahdat. Integrating Microsecond Circuit Switching into the Data Center. In Proc. SIGCOMM, 2013.
20. R. Prasad, M. Jain, and C. Dovrolis. Effects of Interrupt Coalescence on Network Measurements. In Proc. PAM, 2004.
21. A. Rao, A. Legout, Y.-s. Lim, D. Towsley, C. Barakat, and W. Dabbous. Network Characteristics of Video Streaming Traffic. In Proc. CoNEXT, 2011.
22. A. Rasmussen, G. Porter, M. Conley, H. V. Madhyastha, R. N. Mysore, A. Pucher, and A. Vahdat. TritonSort: A Balanced Large-Scale Sorting System. In Proc. NSDI, 2011.
23. V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, , and G. Shi. Design and Implementation of a Consolidated Middlebox Architecture. In Proc. NSDI, 2012.
24. S. Sinha, S. Kandula, and D. Katabi. Harnessing TCPs Burstiness using Flowlet Switching. In Proc. HotNets, 2004.
25. B. Vattikonda, G. Porter, A. Vahdat, and A. C. Snoeren. Practical TDMA for Datacenter Ethernet. In Proc. ACM EuroSys, 2012.
26. D. Wischik. Buffer Sizing Theory for Bursty TCP Flows. In Communications, 2006 International Zurich Seminar on, pages 98--101, 2006.
27. T. Yoshino, Y. Sugawara, K. Inagami, J. Tamatsukuri, M. Inaba, and K. Hiraki. Performance Optimization of TCP/IP over 10 Gigabit Ethernet by Precise Instrumentation. In Proc. SC, 2008.
28. L. Zhang, S. Shenker, and D. D. Clark. Observations on the Dynamics of a Congestion Control Algorithm: The Effects of Two-Way Traffic. In Proc. SIGCOMM, 1991.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Design

    • Measurement

    • Metrics

    • Performance