Opportunistic flooding to improve TCP transmit performance in virtualized clouds

ABSTRACT

Virtualization is a key technology that powers cloud computing platforms such as Amazon EC2. Virtual machine (VM) consolidation, where multiple VMs share a physical host, has seen rapid adoption in practice with increasingly large number of VMs per machine and per CPU core. Our investigations, however, suggest that the increasing degree of VM consolidation has serious negative effects on the VMs' TCP transport performance. As multiple VMs share a given CPU, the scheduling latencies, which can be in the order of tens of milliseconds, substantially increase the typically sub-millisecond round-trip times (RTTs) for TCP connections in a datacenter, causing significant degradation in throughput. In this paper, we propose a light-weight solution called vFlood that (a) allows a TCP sender VM to opportunistically flood the driver domain in the same host, and (b) offloads the VM's TCP congestion control function to the driver domain in order to mask the effects of VM consolidation. Our evaluation of a vFlood prototype on Xen suggests that vFlood substantially improves TCP transmit throughput with minimal per-packet CPU overhead. Further, our application-level evaluation using Apache Olio, a web 2.0 cloud application, indicates a 33% improvement in the number of operations per second.

References

1. Alacritech corporation. http://www.alacritech.com.
2. Apache Olio. http://http://incubator.apache.org/olio/.
3. Chelsio communications. http://www.chelsio.com.
4. CLOC. http://cloc.sourceforge.net.
5. Faban. http://www.opensparc.net/sunsource/faban/www/index.html.
6. The Iperf Benchmark. http://www.noc.ucf.edu/Tools/Iperf/.
7. Linux Networking:TOE. http://www.linuxfoundation. org/collaborate/workgroups/networking/toe.
8. Server Virtualization Landscape. http://events.1105govinfo.com/events/vcg-summit-2010/information/~/media/GIG/GIG%20Events/2010%20Enterprise%20Architecture/Presentations_0/VCG10_3%201_Oltsik%20Bowker.ashx.
9. VMware Knowledge Base article. http://kb.vmware.com/kb/1006143.
10. VMware Tools. http://kb.vmware.com/kb/340.
11. Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., Prabhakar, B., Sengupta, S., and Sridharan, M. Data center TCP (DCTCP). In ACM SIGCOMM (2010).
12. Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., Patterson, D. A., Rabkin, A., Stoica, I., and Zaharia, M. Above the clouds: A Berkeley view of cloud computing. Tech. Rep. UCB/EECS-2009-28, UC Berkeley, 2009.
13. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. Xen and the art of virtualization. In ACM SOSP (2003).
14. Benson, T., Anand, A., Akella, A., and Zhang, M. Understanding data center traffic characteristics. In First ACM Workshop on Research on Enterprise Networking (WREN '09) (2009).
15. Brakmo, L. S., and Peterson, L. L. TCP Vegas: end to end congestion avoidance on a global Internet. IEEE Journal on Selected Areas in Communications 13, 8 (1995).
16. Burtsev, A., Srinivasan, K., Radhakrishnan, P., Bairavasundaram, L. N., Voruganti, K., and Goodson, G. R. Fido: Fast inter-virtual-machine communication for enterprise appliances. In USENIX ATC (2009).
17. Choudhury, A. K., and Hahne, E. L. Dynamic queue length thresholds for shared-memory packet switches. IEEE/ACM Transaction on Networking 6 (1998).
18. Dean, J., and Ghemawat, S. MapReduce: Simplified Data Processing on Large Clusters. In USENIX OSDI (2004).
19. Govindan, S., Nath, A. R., Das, A., Urgaonkar, B., and Sivasubramaniam, A. Xen and Co.: communication-aware CPU scheduling for consolidated Xen-based hosting platforms. In ACM VEE (2007).
20. Gulati, A., Merchant, A., and Varman, P. mClock: Handling throughput variability for hypervisor IO scheduling. In USENIX OSDI'10 (2010).
21. Guo, D., Liao, G., and Bhuyan, L. Performance characterization and cache-aware core scheduling in a virtualized multi-core server under 10GbE. In IEEE IISWC (2009).
22. Gupta, D., Cherkasova, L., Gardner, R., and Vahdat, A. Enforcing performance isolation across virtual machines in Xen. In ACM/USENIX Middleware (2006).
23. Gupta, D., Lee, S., Vrable, M., Savage, S., Snoeren, A. C., Varghese, G., Voelker, G. M., and Vahdat, A. Difference engine: Harnessing memory redundancy in virtual machines. In USENIX OSDI (2008).
24. Ha, S., Rhee, I., and Xu, L. CUBIC: A new TCP-friendly high-speed TCP variant. ACM SIGOPS Operating System Review 42, 5 (2008).
25. Huang, W., Koop, M. J., Gao, Q., and Panda, D. K. Virtual machine aware communication libraries for high performance computing. In ACM/IEEE SC (2007).
26. Jin, C., Wei, D., and Low, S. FAST TCP: Motivation, Architecture, Algorithms, Performance. In IEEE INFOCOM (2004).
27. Kandula, S., Sengupta, S., Greenberg, A., Patel, P., and Chaiken, R. The nature of data center traffic: measurements & analysis. In ACM/USENIX IMC '09 (2009).
28. Kangarlou, A., Gamage, S., Kompella, R. R., and Xu, D. vSnoop: Improving TCP throughput in virtualized environments via acknowledgement offload. In ACM/IEEE SC (2010).
29. Kesavan, M., Gavrilovska, A., and Schwan, K. Differential Virtual Time (DVT): Rethinking I/O service differentiation for virtual machines. In ACM SOCC (2010).
30. Kesavan, M., Gavrilovska, A., and Schwan, K. On disk scheduling in virtual machines. In Second Workshop on I/O Virtualization (WIOV '10) (2010).
31. Kim, K., Kim, C., Jung, S.-I., Shin, H.-S., and Kim, J.-S. Inter-domain socket communications supporting high performance and full binary compatibility on Xen. In ACM VEE (2008).
32. Menon, A., Cox, A. L., and Zwaenepoel, W. Optimizing network virtualization in Xen. In USENIX ATC (2006).
33. Menon, A., Santos, J. R., Turner, Y., Janakiraman, G. J., and Zwaenepoel, W. Diagnosing performance overheads in the Xen virtual machine environment. In ACM VEE (2005).
34. Menon, A., Schubert, S., and Zwaenepoel, W. TwinDrivers: semi-automatic derivation of fast and safe hypervisor network drivers from guest OS drivers. In ACM ASPLOS (2009).
35. Menon, A., and Zwaenepoel, W. Optimizing TCP receive performance. In USENIX ATC (2008).
36. Milos, G., Murray, D. G., Hand, S., and Fetterman, M. A. Satori: Enlightened page sharing. In USENIX ATC (2009).
37. Mogul, J. C. TCP offload is a dumb idea whose time has come. In USENIX HOTOS IX (2003).
38. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., and Zagorodnov, D. The Eucalyptus open-source cloud-computing system. In IEEE/ACM CCGrid (2009).
39. Ongaro, D., Cox, A. L., and Rixner, S. Scheduling I/O in virtual machine monitors. In ACM VEE (2008).
40. Regnier, G., Makineni, S., Illikkal, R., Iyer, R., Minturn, D., Huggahalli, R., Newell, D., Cline, L., and Foong, A. TCP onloading for data center servers. IEEE Computer 37 (2004).
41. Shalev, L., Satran, J., Borovik, E., and Ben-Yehuda, M. IsoStack: Highly efficient network processing on dedicated cores. In USENIX ATC (2010).
42. Sobel, W., Subramanyam, S., Sucharitakul, A., Nguyen, J., Wong, H., Klepchukov, A., Patil, S., Fox, O., and Patterson, D. Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0. In First Workshop on Cloud Computing (CCA) (2008).
43. Vasudevan, V., Phanishayee, A., Shah, H., Krevat, E., Andersen, D. G., Ganger, G. R., Gibson, G. A., and Mueller, B. Safe and effective fine-grained TCP retransmissions for datacenter communication. In ACM SIGCOMM (2009).
44. Wang, J., Wright, K.-L., and Gopalan, K. XenLoop: A transparent high performance inter-vm network loopback. In ACM HPDC (2008).
45. Wood, T., Shenoy, P., Venkataramani, A., and Yousif, M. Black-box and gray-box strategies for virtual machine migration. In USENIX NSDI (2007).
46. Zhang, X., McIntosh, S., Rohatgi, P., and Griffin, J. L. XenSocket: A high-throughput interdomain transport for virtual machines. In ACM/IFIP/USENIX Middleware (2007).

Index Terms

• Networks

  • Network protocols