skip to main content
research-article

B4: experience with a globally-deployed software defined wan

Published: 27 August 2013 Publication History

Abstract

We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic traffic demand that seeks to maximize average bandwidth, and iii) full control over the edge servers and network, which enables rate limiting and demand measurement at the edge.
These characteristics led to a Software Defined Networking architecture using OpenFlow to control relatively simple switches built from merchant silicon. B4's centralized traffic engineering service drives links to near 100% utilization, while splitting application flows among multiple paths to balance capacity against application priority/demands. We describe experience with three years of B4 production deployment, lessons learned, and areas for future work.

References

[1]
Al-Fares, M., Loukissas, A., and Vahdat, A. A Scalable, Commodity Data Center Network Architecture. In Proc. SIGCOMM (New York, NY, USA, 2008), ACM.
[2]
Allalouf, M., and Shavitt, Y. Centralized and Distributed Algorithms for Routing and Weighted Max-Min Fair Bandwidth Allocation. IEEE/ACM Trans. Networking 16, 5 (2008), 1015--1024.
[3]
Aukia, P., Kodialam, M., Koppol, P. V., Lakshman, T. V., Sarin, H., and Suter, B. RATES: A Server for MPLS Traffic Engineering. IEEE Network Magazine 14, 2 (March 2000), 34--41.
[4]
Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and Swallow, G. RSVP-TE: Extensions to RSVP for LSP Tunnels. RFC 3209, IETF, United States, 2001.
[5]
Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and McManus, J. Requirements for Traffic Engineering Over MPLS. RFC 2702, IETF, 1999.
[6]
Caesar, M., Caldwell, D., Feamster, N., Rexford, J., Shaikh, A., and van der Merwe, K. Design and Implementation of a Routing Control Platform. In Proc. of NSDI (April 2005).
[7]
Casado, M., Freedman, M. J., Pettit, J., Luo, J., McKeown, N., and Shenker, S. Ethane: Taking Control of the Enterprise. In Proc. SIGCOMM (August 2007).
[8]
Casado, M., Garfinkel, T., Akella, A., Freedman, M. J., Boneh, D., McKeown, N., and Shenker, S. SANE: A Protection Architecture for Enterprise Networks. In Proc. of Usenix Security (August 2006).
[9]
Chandra, T. D., Griesemer, R., and Redstone, J. Paxos Made Live: an Engineering Perspective. In Proc. of the ACM Symposium on Principles of Distributed Computing (New York, NY, USA, 2007), ACM, pp. 398--407.
[10]
Choi, T., Yoon, S., Chung, H., Kim, C., Park, J., Lee, B., and Jeong, T. Design and Implementation of Traffic Engineering Server for a Large-Scale MPLS-Based IP Network. In Revised Papers from the International Conference on Information Networking, Wireless Communications Technologies and Network Applications-Part I (London, UK, UK, 2002), ICOIN '02, Springer-Verlag, pp. 699--711.
[11]
Curtis, A. R., Mogul, J. C., Tourrilhes, J., Yalagandula, P., Sharma, P., and Banerjee, S. DevoFlow: Scaling Flow Management for High-Performance Networks. In Proc. SIGCOMM (2011), pp. 254--265.
[12]
Danna, E., Hassidim, A., Kaplan, H., Kumar, A., Mansour, Y., Raz, D., and Segalov, M. Upward Max Min Fairness. In INFOCOM (2012), pp. 837--845.
[13]
Danna, E., Mandal, S., and Singh, A. A Practical Algorithm for Balancing the Max-min Fairness and Throughput Objectives in Traffic Engineering. In Proc. INFOCOM (March 2012), pp. 846--854.
[14]
Elwalid, A., Jin, C., Low, S., and Widjaja, I. MATE: MPLS Adaptive Traffic Engineering. In Proc. IEEE INFOCOM (2001), pp. 1300--1309.
[15]
Farrington, N., Rubow, E., and Vahdat, A. Data Center Switch Architecture in the Age of Merchant Silicon. In Proc. Hot Interconnects (August 2009), IEEE, pp. 93--102.
[16]
Fortz, B., Rexford, J., and Thorup, M. Traffic Engineering with Traditional IP Routing Protocols. IEEE Communications Magazine 40 (2002), 118--124.
[17]
Fortz, B., and Thorup, M. Increasing Internet Capacity Using Local Search. Comput. Optim. Appl. 29, 1 (October 2004), 13--48.
[18]
Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. VL2: A Scalable and Flexible Data Center Network. In Proc. SIGCOMM (August 2009).
[19]
Greenberg, A., Hjalmtysson, G., Maltz, D. A., Myers, A., Rexford, J., Xie, G., Yan, H., Zhan, J., and Zhang, H. A Clean Slate 4D Approach to Network Control and Management. SIGCOMM CCR 35, 5 (2005), 41--54.
[20]
Greenberg, A., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. Towards a Next Generation Data Center Architecture: Scalability and Commoditization. In Proc. ACM workshop on Programmable Routers for Extensible Services of Tomorrow (2008), pp. 57--62.
[21]
Gude, N., Koponen, T., Pettit, J., Pfaff, B., Casado, M., McKeown, N., and Shenker, S. NOX: Towards an Operating System for Networks. In SIGCOMM CCR (July 2008).
[22]
He, J., and Rexford, J. Toward Internet-wide Multipath Routing. IEEE Network Magazine 22, 2 (March 2008), 16--21.
[23]
Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., and Wattenhofer, R. Have Your Network and Use It Fully Too: Achieving High Utilization in Inter-Datacenter WANs. In Proc. SIGCOMM (August 2013).
[24]
Kandula, S., Katabi, D., Davie, B., and Charny, A. Walking the Tightrope: Responsive Yet Stable Traffic Engineering. In Proc. SIGCOMM (August 2005).
[25]
Kipp, S. Bandwidth Growth and the Next Speed of Ethernet. Proc. North American Network Operators Group (October 2012).
[26]
Koponen, T., Casado, M., Gude, N., Stribling, J., Poutievski, L., Zhu, M., Ramanathan, R., Iwata, Y., Inoue, H., Hama, T., and Shenker, S. Onix: a Distributed Control Platform for Large-scale Production Networks. In Proc. OSDI (2010), pp. 1--6.
[27]
Lakshman, T., Nandagopal, T., Ramjee, R., Sabnani, K., and Woo, T. The Softrouter Architecture. In Proc. HotNets (November 2004).
[28]
McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J. OpenFlow: Enabling Innovation in Campus Networks. SIGCOMM CCR 38, 2 (2008), 69--74.
[29]
Medina, A., Taft, N., Salamatian, K., Bhattacharyya, S., and Diot, C. Traffic Matrix Estimation: Existing Techniques and New Directions. In Proc. SIGCOMM (New York, NY, USA, 2002), ACM, pp. 161--174.
[30]
Nascimento, M. R., Rothenberg, C. E., Salvador, M. R., and Magalh\ aes, M. F. QuagFlow: Partnering Quagga with OpenFlow (Poster). In Proc. SIGCOMM (2010), pp. 441--442.
[31]
OpenFlow Specification. http://www.openflow.org/wp/documents/.
[32]
Rothenberg, C. E., Nascimento, M. R., Salvador, M. R., Corrêa, C. N. A., Cunha de Lucena, S., and Raszuk, R. Revisiting Routing Control Platforms with the Eyes and Muscles of Software-defined Networking. In Proc. HotSDN (2012), pp. 13--18.
[33]
Roughan, M., Thorup, M., and Zhang, Y. Traffic Engineering with Estimated Traffic Matrices. In Proc. IMC (2003), pp. 248--258.
[34]
Scoglio, C., Anjali, T., de Oliveira, J. C., Akyildiz, I. F., and UhI, G. TEAM: A Traffic Engineering Automated Manager for DiffServ-based MPLS Networks. Comm. Mag. 42, 10 (October 2004), 134--145.
[35]
Sherwood, R., Gibb, G., Yap, K.-K., Appenzeller, G., Casado, M., McKeown, N., and Parulkar, G. FlowVisor: A Network Virtualization Layer. Tech. Rep. OPENFLOW-TR-2009--1, OpenFlow, October 2009.
[36]
Suchara, M., Xu, D., Doverspike, R., Johnson, D., and Rexford, J. Network Architecture for Joint Failure Recovery and Traffic Engineering. In Proc. ACM SIGMETRICS (2011), pp. 97--108.
[37]
Thaler, D. Multipath Issues in Unicast and Multicast Next-Hop Selection. RFC 2991, IETF, 2000.
[38]
Wang, H., Xie, H., Qiu, L., Yang, Y. R., Zhang, Y., and Greenberg, A. COPE: Traffic Engineering in Dynamic Networks. In Proc. SIGCOMM (2006), pp. 99--110.
[39]
Xu, D., Chiang, M., and Rexford, J. Link-state Routing with Hop-by-hop Forwarding Can Achieve Optimal Traffic Engineering. IEEE/ACM Trans. Netw. 19, 6 (December 2011), 1717--1730.
[40]
Yu, M., Rexford, J., Freedman, M. J., and Wang, J. Scalable flow-based networking with DIFANE. In Proc. SIGCOMM (2010), pp. 351--362.

Cited By

View all
  • (2025)Verifying Network-level Properties for Large-scale Networks with Header Transformations in RealtimeJournal of Information Processing10.2197/ipsjjip.33.4133(41-54)Online publication date: 2025
  • (2025)Rethinking Cost-Efficient VM Scheduling on Public Edge Platforms: A Service Provider’s PerspectiveIEEE Transactions on Mobile Computing10.1109/TMC.2024.348808224:3(1846-1858)Online publication date: Mar-2025
  • (2025)Analyzing the Benefits of Optical Topology Programming for Mitigating Link-Flood DDoS AttacksIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.339118822:1(146-163)Online publication date: 1-Jan-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGCOMM Computer Communication Review
ACM SIGCOMM Computer Communication Review  Volume 43, Issue 4
October 2013
595 pages
ISSN:0146-4833
DOI:10.1145/2534169
Issue’s Table of Contents
  • cover image ACM Conferences
    SIGCOMM '13: Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
    August 2013
    580 pages
    ISBN:9781450320566
    DOI:10.1145/2486001
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 August 2013
Published in SIGCOMM-CCR Volume 43, Issue 4

Check for updates

Author Tags

  1. centralized traffic engineering
  2. openflow
  3. routing
  4. software- defined networking
  5. wide-area networks

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,282
  • Downloads (Last 6 weeks)130
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Verifying Network-level Properties for Large-scale Networks with Header Transformations in RealtimeJournal of Information Processing10.2197/ipsjjip.33.4133(41-54)Online publication date: 2025
  • (2025)Rethinking Cost-Efficient VM Scheduling on Public Edge Platforms: A Service Provider’s PerspectiveIEEE Transactions on Mobile Computing10.1109/TMC.2024.348808224:3(1846-1858)Online publication date: Mar-2025
  • (2025)Analyzing the Benefits of Optical Topology Programming for Mitigating Link-Flood DDoS AttacksIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2024.339118822:1(146-163)Online publication date: 1-Jan-2025
  • (2025)Maintaining Predictable Traffic Engineering Performance Under Controller Failures for Software-Defined WANsIEEE Journal on Selected Areas in Communications10.1109/JSAC.2025.352881443:2(524-536)Online publication date: Feb-2025
  • (2025)SDN-Based Service Function Chaining in Integrated Terrestrial and LEO Satellite-Based Space InternetIEEE Journal on Selected Areas in Communications10.1109/JSAC.2025.352880743:2(537-550)Online publication date: Feb-2025
  • (2025)Using spanners to improve network performanceComputer Networks10.1016/j.comnet.2024.110976257(110976)Online publication date: Feb-2025
  • (2025)Safe load balancing in software-defined-networkingComputer Communications10.1016/j.comcom.2024.107985229(107985)Online publication date: Jan-2025
  • (2025)AI-enabled routing in next generation networks: A surveyAlexandria Engineering Journal10.1016/j.aej.2025.01.095120(449-474)Online publication date: May-2025
  • (2025)BI-TE: achieving GNN-based bandwidth indistinguishable traffic engineering in multi-domain SDNFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-40551-219:11Online publication date: 1-Nov-2025
  • (2025)A comprehensive survey on software-defined wide area network (SD-WAN): principles, opportunities and future challengesThe Journal of Supercomputing10.1007/s11227-024-06718-181:1Online publication date: 1-Jan-2025
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media