Hierarchical SDN for the hyper-scale, hyper-elastic data center and cloud

ABSTRACT

With the explosive growth in the demand for cloud services, the Data Center and Data Center Interconnect have to achieve hyper-scale and provide unprecedented elasticity and resource availability. The underlay network infrastructure has to scale to support tens of millions of physical endpoints at low cost; the virtualized overlay layer has to scale to millions of Virtual Networks connecting hundreds of millions of Virtual Machines (VMs) and Virtualized Network Functions (VNFs), and provide seamless VM and VNF mobility.

In this paper, we present Hierarchical SDN (HSDN), an architectural solution that achieves hyper scale using surprisingly small forwarding tables in the network nodes. HSDN introduces a new paradigm for the forwarding and control planes, in that all paths in the network are pre-established in the forwarding tables and the labels identify entire paths rather than simply destinations. These properties of HSDN dramatically simplify establishing tunnels, and thus enable optimal handling of both ECMP and any-to-any end-to-end TE, which in turn yields extremely high network utilization with small buffers in the switches. The pre-established tunnels make HSDN the ideal underlay infrastructure to enable seamless and lossless VM and VNF overlay mobility, and achieve excellent elasticity.

HSDN is suitable for a full SDN implementation, using a scalable SDN controller to configure all forwarding tables in the network nodes and in the endpoints, as well as a hybrid approach, using conventional routing protocols in conjunction with a SDN controller.

References

1. A. Greenberg, G. Hjalmtysson, D. Maltz, A. Myers, J. Rexford, G. Xie, H. Yan, J. Zhan, and H. Zhang, "A Clean Slate 4D Approach to Network Control and Management," ACM SIGCOMM Computer Comm. Review, October 2005.
2. S. Shenker, M. Casado, T. Koponen, and N. McKeown, "The Future of Networking, the Past of Protocols," in Open Network Summit 2013, Santa Clara, CA, April 2013.
3. M. Casado, M. Freedman, J. Pettit, J. Luo, N. Gude, N. McKeown, and S. Shenker, "Rethinking Enterprise Network Control," IEEE/ACM Trans. on Networking, August 2009.
4. E. Rosen, A. Viswanathan, and R. Callon, "Multiprotocol Label Switching Architecture," IETF RFC 3031, January 2001.
5. L. Fang, V. Gill, D. Bansal, F. Chiussi, C. Ramachandran, S. Davari, L. Dunbar, B Gafni, A. Qu, J. Tantsura, Y. Rekhter, E. Aries, D. Voyer, W. Wang, H. Shah, and R. Krishnan, "MPLS-Based Hierarchical SDN for Hyper-Scale DC/Cloud," IETF draft-fang-hsdn-for-hsdc-02 (work in progress), March 2015.
6. E. Rosen, D. Tappan, G. Fedorkow, Y. Rekhter, D. Farinacci, T. Li, and A. Conta, "MPLS Label Stack Encoding," IETF RFC 3032, January 2001.
7. Y. Rekhter and E. Rosen, "Carrying Label Information in BGP-4," IETF RFC 3107, May 2001.
8. L. Fang, C. Ramachandran, F. Chiussi, and Y. Rekhter, "BGP-LU for HSDN Label Distribution," IETF draft-fangidr-bgplu-for-hsdn-00 (work in progress), March 2015.
9. C.-Y. Hong, S. Kandula, R. Mahajan, M. Zhang, V. Gill, M. Nanduri, and R. Wattenhofer, "Achieving High Utilization with Software-Driven WAN," in ACM SIGCOMM 2013, Hong Kong, August 2013.
10. M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat, "Hedera: Dynamic flow scheduling for data center networks," in Symposium on Networked Systems Design and Implementation 2010, San Jose, CA, April 2010.
11. D. Awduche, L. Berger, D. Gan, T. Li, V. Srinivasan, and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels," IETF RFC 3209, December 2001.
12. S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Holzle, S. Stuart, and A. Vahdat, "B4: Experience with a Globally-Deployed Software Defined WAN," in ACM SIGCOMM 2013, Hong Kong, August 2013.
13. M. Alizadeh and T. Edsall, "On the Data Path Performance of Leaf-Spine Datacenter Fabrics," in 2013 IEEE Annual Symposium on High-Performance Interconnects, San Jose, CA, August 2013.
14. A. Greenberg, J. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. Maltz, P. Patel, and S. Sengupta, "VL2: A Scalable and Flexible Data Center Network," in ACM SIGCOMM 2009, Barcelona, Spain, August 2009.
15. H. Qi, M. Shiraz, J. Liu, A. Gani, Z. Abdul Rahman, and T. Altameem, "Data center network architecture in cloud computing: review, taxonomy, and open research issues," J. of Zhejiang University Science, September 2014, http://link.springer.com/article/10.1631%2Fjzus.C1400013.
16. E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)," IETF RFC 4364, February 2006.
17. M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks," IETF RFC 7348, August 2014.
18. P. Garg and Y. Wang, Editors, "NVGRE: Network Virtualization using Generic Routing Encapsulation," IETF draft-sridharan-virtualization-nvgre-07 (work in progress), April 2015.
19. E. Osborne and A. Simha, "Traffic Engineering with MPLS," Cisco Press, 2002.
20. M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian. "Fabric: A Retrospective on Evolving SDN," in HotSDN 2012, Helsinki, Finland, August 2012.
21. J. McCauley A. Panda, M. Casado, T. Koponen, and S. Shenker, "Extending SDN to Large-Scale Networks," in Open Network Summit 2013, Santa Clara, CA, April 2013.
22. M. Al-Fares, A. Loukissas, and A. Vahdat, "A Scalable, Commodity Data Center Network Architecture," in ACM SIGCOMM 2008, Seattle, WA, August 2008.
23. C. Guo, H. Wu, K. Tan, L Shi, Y. Zhang, and S. Lu, "DCell: A Scalable and Fault-Tolerant Network Structure for Data Centers," in ACM SIGCOMM 2008, Seattle, WA, August 2008.
24. Y. Fu, J. Bi, K. Gao, Z. Chen, J. Wu, and B. Hao, "Orion: A Hybrid Hierarchical Control Plane of Software-Defined Networking for Large-Scale Networks," in CoolSDN 2014, Raleigh, NC, October 2014.
25. M. Moradi, W. Wu, L. Li, and Z. Mao, "SoftMow: Recursive and Reconfigurable Cellular WAN Architecture," in GoNEXT 2014, Sydney, Australia, December 2014.
26. M. Yu, J. Rexford, M. Freedan, and J. Wang, "Scalable Flow-Based Networking with DIFANE," in ACM SIGCOMM 2010, New Delhi, India, August 2010.
27. T. Benson, A. Anand, A. Akella, and M. Zhang, "MicroTE: Fine Grained Traffic Engineering for Data Centers," in ACM CoNEXT 2011, Tokyo, Japan, December 2011.
28. T. Benson, A. Akella, and D. Maltz, "Network Traffic Characteristics of Data Centers in the Wild," in IMC 2010, Melbourne, Australia, November 2010.
29. S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken, "The Nature of Datacenter Traffic: Measurements & Analysis," in IMC 2009, Chicago, IL, November 2009.
30. Y. Qiao, Z. Hu, and J. Luo, "Efficient Traffic Matrix Estimation for Data Center Networks," in IFIP Networking Conf. 2013, Brooklyn, NY, May 2013.
31. C. Filsfils, S. Previdi, B. Decraene, S. Litkowski, and R. Shakir, "Segment Routing Architecture," IETF draft-ietf-spring-segment-routing-03 (work in progress), May 2015.

Index Terms

• Networks

  • Network architectures