skip to main content
10.1145/3127479.3132252acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
research-article
Public Access

UNO: uniflying host and smart NIC offload for flexible packet processing

Published: 24 September 2017 Publication History

Abstract

Increasingly, smart Network Interface Cards (sNICs) are being used in data centers to offload networking functions (NFs) from host processors thereby making these processors available for tenant applications. Modern sNICs have fully programmable, energy-efficient multi-core processors on which many packet processing functions, including a full-blown programmable switch, can run. However, having multiple switch instances deployed across the host hypervisor and the attached sNICs makes controlling them difficult and data plane operations more complex.
This paper proposes a generalized SDN-controlled NF offload architecture called UNO. It can transparently offload dynamically selected host processors' packet processing functions to sNICs by using multiple switches in the host while keeping the data centerwide network control and management planes unmodified. UNO exposes a single virtual control plane to the SDN controller and hides dynamic NF offload behind a unified virtual management plane. This enables UNO to make optimal use of host's and sNIC's combined packet processing capabilities with local optimization based on locally observed traffic patterns and resource consumption, and without central controller involvement. Experimental results based on a real UNO prototype in realistic scenarios show promising results: it can save processing worth up to 8 CPU cores, reduce power usage by up to 2x, and reduce the control plane overhead by more than 50%.

References

[1]
Accolade ANIC. https://accoladetechnology.com/whitepapers/ANIC-Features-Overview.pdf.
[2]
Cavium LiquidIO. http://www.cavium.com/pdfFiles/LiquidIO_Server_Adapters_PB_Rev1.0.pdf.
[3]
Data Center Market Trends. http://www.te.com/content/dam/te-com/documents/broadband-network-solutions/global/data-center/brochures/presentation-data-center-market-trends.pdf.
[4]
Emerging Smart NIC Technology. http://www.csit.qub.ac.uk/News/Events/Belfast-2016-6th-Cyber-Security-Summit/PDFs/Filetoupload,631658,en.pdf.
[5]
Floodlight. http://www.projectfloodlight.org/floodlight/.
[6]
How to Port Open vSwitch to New Software or Hardware. http://openvswitch.org/support/dist-docs-2.5/PORTING.md.html.
[7]
Intel Advanced Encryption Standard (Intel AES) Instructions Set - Rev 3.01. https://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set.
[8]
Intel Clear Containers: A Breakthrough Combination of Speed and Workload Isolation. https://clearlinux.org/sites/default/files/vmscontainers_wp_v5.pdf.
[9]
Intel Gigabit Server Adapters. http://ark.intel.com/products/family/46829.
[10]
Intel QuickAssist Adapter Family for Servers. http://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-for-servers.html.
[11]
Max-flow min-cut theorem. https://en.wikipedia.org/wiki/Max-flow_min-cut_theorem.
[12]
Mellanox BlueField. http://www.mellanox.com/related-docs/npu-multicore-processors/PB_Bluefield_SoC.pdf.
[13]
Mellanox ConnectX-4. http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_VPI_Card.pdf.
[14]
Mellanox ConnectX-5. http://www.mellanox.com/related-docs/user_manuals/ConnectX-5_VPI_Card.pdf.
[15]
Minimum k-cut. https://en.wikipedia.org/wiki/Minimum_k-cut.
[16]
nDPI. http://www.ntop.org/products/deep-packet-inspection/ndpi/.
[17]
Netronome Agilio vRouter. https://netronome.com/media/redactor_files/SB_Netronome_Juniper_vRouter.pdf.
[18]
OpenDaylight. https://www.opendaylight.org.
[19]
OpenStack. https://www.openstack.org.
[20]
OpenStack Compute API. https://developer.openstack.org/api-ref/compute/.
[21]
OPNFV. https://www.opnfv.org.
[22]
PRADS - Passive Real-time Asset Detection System. https://gamelinux.github.io/prads/.
[23]
Processor Counter Monitor. https://github.com/opcm/pcm.
[24]
Programming Intel QuickAssist Technology Hardware Accelerators for Optimal Performance. https://01.org/sites/default/files/page/332125_002_0.pdf.
[25]
Putting Smart NICs in White Boxes. https://www.sdxcentral.com/articles/analysis/nics-white-boxes/2016/11/.
[26]
SD-WAN. https://en.wikipedia.org/wiki/SD-WAN.
[27]
Setting up the Python API of CPLEX. http://www.ibm.com/support/knowledgecenter/SSSA5P_12.5.1/ilog.odms.cplex.help/CPLEX/GettingStarted/topics/set_up/Python_setup.html.
[28]
TILEncore-Gx36. http://www.mellanox.com/related-docs/prod_multi_core/PB_TILEncore-Gx36.pdf.
[29]
Tilera Rescues CPU Cycles with Network Coprocessors. https://www.enterprisetech.com/2013/10/16/tilera-free-expensive-cpu-cycles-network-coprocessors/.
[30]
VMware. Data Center Micro-Segmentation. http://blogs.vmware.com/networkvirtualization/files/2014/06/VMware-SDDC-Micro-Segmentation-White-Paper.pdf.
[31]
Watts Up Meter. https://www.wattsupmeters.com.
[32]
TILE Processor Architecture Overview for the TILE-Gx Series. Technical report, Mellanox, 2012. Doc. No. UG130.
[33]
OpenFlow Switch Specification 1.5.0. Open Network Foundation, 2014.
[34]
A. Al-Shabibi et al. OpenVirteX: Make Your Virtual SDNs Programmable. In Proc. ACM HotSDN, 2014.
[35]
S. P. Antoine Kaufmann and N. K. Sharma. High Performance Packet Processing with FlexNIC. In Proc. ASPLOS, 2016.
[36]
H. Ballani et al. Enabling End-host Network Functions. In Proc. ACM SIGCOMM, 2015.
[37]
A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion. IX: A Protected Dataplane Operating System for High Throughput and Low Latency. In Proc. USENIX OSDI, 2014.
[38]
M. Blott and K. Vissers. Dataflow Architectures for 10Gbps Line-rate Key-value-Stores. In Proc. IEEE Hot Chips 25 Symposium, 2013.
[39]
P. Bosshart et al. P4: Programming Protocol-Independent Packet Processors. ACM SIGCOMM Computer Communication Review, 44(3), 2014.
[40]
Z. Bozakov and P. Papadimitriou. AutoSlice: Automated and Scalable Slicing for Software-Defined Networks. In Proc. ACM CoNEXT, 2012.
[41]
M. Casado, T. Koponen, S. Shenker, and A. Tootoonchian. Fabric: A Retrospective on Evolving SDN. In Proc. ACM HotSDN, 2012.
[42]
H. Chang, S. Mukherjee, L. Wang, T. Lakshman, Y. Le, A. Akella, and M. Swift. UNO: Unifying Host and Smart NIC Offload for Flexible Packet Processing. Technical Report ITD-16-56788B, Nokia, 2016.
[43]
Cisco. Data Center Microsegmentation: Enhance Security for Data Center Traffic. http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-732943.html.
[44]
E. Cuervo et al. MAUI: Making Smartphones Last Longer with Code Offload. In Proc. ACM MobiSys, 2010.
[45]
H. T. Dang et al. Network Hardware-Accelerated Consensus. In USI Technical Report Series in Informatics, 2016.
[46]
R. R. David F. Bacon and S. Shukla. FPGA Programming for the Masses. ACM QUEUE, 11(2), 2013.
[47]
W. Dietz, J. Cranmer, N. Dautenhahn, and V. Adve. Slipstream: Automatic Interprocess Communication Optimization. In Proc. USENIX ATC, 2015.
[48]
S. K. Fayazbakhsh, L. Chiang, V. Sekar, M. Yu, and J. C. Mogul. Enforcing Network-Wide Policies in the Presence of Dynamic Middlebox Actions using FlowTags. In Proc. USENIX NSDI, 2014.
[49]
D. Firestone. SmartNIC: Accelerating Azure's Network with FPGAs on OCS Servers. Open Compute Project, 2016.
[50]
X. Ge, Y. Liu, D. H. Du, L. Zhang, H. Guan, J. Chen, Y. Zhao, and X. Hu. OpenANFV: Accelerating Network Function Virtualization with a Consolidated Framework in OpenStack. In Proc. ACM SIGCOMM, 2014.
[51]
A. Gember, P. Prabhu, Z. Ghadiyali, and A. Akella. Toward Software-defined Middlebox Networking. In Proc. ACM HotNets-XI, 2012.
[52]
A. Gember-Jacobson et al. OpenNF: Enabling Innovation in Network Function Control. ACM SIGCOMM Computer Communication Review, 44(4), 2015.
[53]
B. Grot et al. Optimizing Data-Center TCO with Scale-Out Processors. IEEE Micro, 32(5), 2012.
[54]
B. Han, V. Gopalakrishnan, L. Ji, and S. Lee. Network Functions Virtualization: Challenges and Opportunities for Innovations. IEEE Communication Magazine, 53(2), 2015.
[55]
S. Han, K. Jang, A. Panda, S. Palkar, D. Han, and S. Ratnasamy. SoftNIC: A Software NIC to Augment Hardware. Technical Report UCB/EECS-2015-155, University of California, Berkeley, 2015.
[56]
A. Holt et al. Cloud Computing Takes Off. https://www.morganstanley.com/views/perspectives/cloud_computing.pdf. Morgan Stanley.
[57]
M. Honda, F. Huici, G. Lettieri, and L. Rizzo. mSwitch: A Highly-Scalable, Modular Software Switch. In Proc. ACM SOSR, 2015.
[58]
J. Hwang, K. K. Ramakrishnan, and T. Wood. NetVM: High Performance and Flexible Networking using Virtualization on Commodity Platforms. In Proc. USENIX NSDI, 2014.
[59]
Z. Istvan, D. Sidler, G. Alonso, and M. Vukolic. Consensus in a Box: Inexpensive Coordination in Hardware. In Proc. USENIX NSDI, 2016.
[60]
E. J. Jackson, M. Walls, A. Panda, J. Pettit, B. Pfaff, J. Rajahalme, T. Koponen, and S. Shenker. SoftFlow: A Middlebox Architecture for Open vSwitch. In Proc. USENIX ATC, 2016.
[61]
M. Kablan, A. Alsudais, E. Keller, and F. Le. Stateless Network Functions: Breaking the Tight Coupling of State and Processing. In Proc. USENIX NSDI, 2017.
[62]
N. Kang, Z. Liu, J. Rexford, and D. Walker. Optimizing the One Big Switch Abstraction in Software-Defined Networks. In Proc. ACM CoNEXT, 2013.
[63]
Y. Kanizo, D. Hay, and I. Keslassy. Palette: Distributing Tables in Software-Defined Networks. In Proc. ACM CoNEXT, 2013.
[64]
N. P. Katta, J. Rexford, and D. Walker. Incremental Consistent Updates. In Proc. ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, 2013.
[65]
S. Kent. IP Encapsulating Security Payload (ESP). RFC 4303, 2005.
[66]
A. Khrabrov and E. de Lara. Accelerating Complex Data Transfer for Cluster Computing. In Proc. USENIX HotCloud, 2016.
[67]
Kindervag, J. Build Security Into Your Network's DNA: The Zero Trust Network Architecture.
[68]
S. Larsen and B. Lee. Platform IO DMA Transaction Acceleration. In Proc. ACM Workshop on Characterizing Applications for Heterogeneous Exascale Systems, 2011.
[69]
J. Li, E. Michael, N. K. Sharma, A. Szekeres, and D. R. K. Ports. Just say NO to Paxos Overhead: Replacing Consensus with Network Ordering. In Proc. USENIX OSDI, 2016.
[70]
K. Lim et al. Thin Servers with Smart Pipes: Designing SoC Accelerators for Memcached. In Proc. ISCA, 2013.
[71]
Y. Luo, E. Murray, and T. L. Ficarra. Accelerated Virtual Switching with Programmable NICs for Scalable Data Center Networking. In Proc. ACM VISA, 2010.
[72]
H. Mekky, F. Hao, S. Mukherjee, Z.-L. Zhang, and T. Lakshman. Application-aware Data Plane Processing in SDN. In Proc. ACM HotSDN, 2014.
[73]
M. Moshref, M. Yu, A. Sharma, and R. Govindan. vCRIB: Virtualized Rule Management in the Cloud. In Proc. USENIX HotCloud, 2012.
[74]
M. Moshref, M. Yu, A. Sharma, and R. Govindan. Scalable Rule Management for Data Centers. In Proc. USENIX NSDI, 2013.
[75]
J. Nam, M. Jamshed, B. Choi, D. Han, and K. Park. Scaling the Performance of Network Intrusion Detection with Many-core Processors. In Proc. ACM/IEEE ANCS, 2015.
[76]
S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Ratnasamy, L. Rizzo, and S. Shenker. E2: A Framework for NFV Applications. In Proc. ACM SOSP, 2015.
[77]
Palo Alto Networks. Getting Started With a Zero Trust Approach to Network Security. https://www.paloaltonetworks.com/resources/whitepapers/zero-trust-network-security.html.
[78]
T. Park, Y. Kim, and S. Shin. UNISAFE: A Union of Security Actions for Software Switches. In Proc. SDN-NFV Security, 2016.
[79]
S. Peter, J. Li, I. Zhang, D. R. K. Ports, D. Woos, A. Krishnamurthy, T. Anderson, and T. Roscoe. Arrakis: The Operating System is the Control Plane. In Proc. USENIX OSDI, 2014.
[80]
J. Pettit. Open vSwitch and the Intelligent Edge. In Proc. OpenStack Summit Atlanta, 2014.
[81]
B. Pfaff et al. The Design and Implementation of Open vSwitch. In Proc. USENIX NSDI, 2015.
[82]
Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu. SIMPLE-fying Middlebox Policy Enforcement Using SDN. In Proc. ACM SIGCOMM, 2013.
[83]
S. Radhakrishnan, Y. Geng, V. Jeyakumar, A. Kabbani, G. Porter, and A. Vahdat. SENIC: Scalable NIC for End-Host Rate Limiting. In Proc. USENIX NSDI, 2014.
[84]
B. Raghavan et al. Software-Defined Internet Architecture: Decoupling Architecture from Infrastructure. In Proc. ACM HotNets-XI, 2012.
[85]
K. K. Ram, A. L. Cox, M. Chadha, and S. Rixner. Hyper-switch: A scalable software virtual switching architecture. In Proc. USENIX ATC, 2013.
[86]
K. K. Ram et al. sNICh: Efficient Last Hop Networking in the Data Center. In Proc. ACM/IEEE ANCS, 2010.
[87]
L. Rizzo, P. Valente, G. Lettieri, and V. Maffione. PSPAT: software packet scheduling at hardware speed. Preprint, 2016.
[88]
G. Sabin and M. Rashti. Security Offload Using the SmartNIC, A Programmable 10 Gbps Ethernet NIC. In Proc. Aerospace and Electronics Conference, 2015.
[89]
V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi. Design and Implementation of a Consolidated Middlebox Architecture. In Proc. USENIX NSDI, 2012.
[90]
A. Shaikh, J. Rexford, and K. G. Shin. Load-Sensitive Routing of Long-Lived IP Flows. In Proc. ACM SIGCOMM, 1999.
[91]
J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, and V. Sekar. Making Middleboxes Someone else's Problem: Network Processing As a Cloud Service. In Proc. ACM SIGCOMM, 2012.
[92]
R. Sherwood et al. FlowVisor: A Network Virtualization Layer. In OpenFlow Switch Consortium, 2009.
[93]
P. Shinde, A. Kaufmann, T. Roscoe, and S. Kaestle. We need to talk about NICs. In Proc. USENIX HotOS, 2013.
[94]
D. Sturgeon. HW Acceleration of Memcached. In Proc. Flash Memory Summit, 2014.
[95]
A. Tootoonchian and Y. Ganjali. HyperFlow: A Distributed Control Plane for OpenFlow. In Proc. Internet Network Management Conference on Research on Enterprise Networking, 2010.
[96]
A. Wang, Y. Guo, F. Hao, T. V. Lakshman, and S. Chen. UMON: Flexible and Fine Grained Traffic Monitoring in Open vSwitch. In Proc. ACM CoNEXT, 2015.
[97]
Z. Wang, K. Liu, Y. Shen, J. Y. B. Lee, M. Chen, and L. Zhang. Intra-host Rate Control with Centralized Approach. In Proc. IEEE International Conference on Cluster Computing, 2016.
[98]
Y. Weinsberg, D. Dolev, P. Wyckoff, and T. Anker. Accelerating Distributed Computing Applications Using a Network Offloading Framework. In Proc. IEEE Parallel and Distributed Processing Symposium, 2007.
[99]
M. Yu, J. Rexford, M. J. Freedman, and J. Wang. Scalable Flow-Based Networking with DIFANE. In Proc. ACM SIGCOMM, 2010.

Cited By

View all
  • (2025)Performance Prediction of On-NIC Network Functions with Multi-Resource Contention and Traffic AwarenessProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707232(828-842)Online publication date: 30-Mar-2025
  • (2024)OSMOSISProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692007(247-263)Online publication date: 10-Jul-2024
  • (2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
  • Show More Cited By

Index Terms

  1. UNO: uniflying host and smart NIC offload for flexible packet processing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SoCC '17: Proceedings of the 2017 Symposium on Cloud Computing
    September 2017
    672 pages
    ISBN:9781450350280
    DOI:10.1145/3127479
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 September 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. networking and SDNs
    2. virtualization and containers

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    SoCC '17
    Sponsor:
    SoCC '17: ACM Symposium on Cloud Computing
    September 24 - 27, 2017
    California, Santa Clara

    Acceptance Rates

    Overall Acceptance Rate 169 of 722 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)426
    • Downloads (Last 6 weeks)50
    Reflects downloads up to 02 Mar 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Performance Prediction of On-NIC Network Functions with Multi-Resource Contention and Traffic AwarenessProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707232(828-842)Online publication date: 30-Mar-2025
    • (2024)OSMOSISProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692007(247-263)Online publication date: 10-Jul-2024
    • (2024)DPC: DPU-accelerated High-Performance File System ClientProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673123(63-72)Online publication date: 12-Aug-2024
    • (2024)μMon: Empowering Microsecond-level Network Monitoring with WaveletsProceedings of the ACM SIGCOMM 2024 Conference10.1145/3651890.3672236(274-290)Online publication date: 4-Aug-2024
    • (2024)Hoda: a High-performance Open vSwitch Dataplane with Multiple Specialized Data PathsProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629564(82-98)Online publication date: 22-Apr-2024
    • (2024)MTDA: Efficient and Fair DPU Offloading Method for Multiple TenantsIEEE Transactions on Services Computing10.1109/TSC.2024.3433588(1-14)Online publication date: 2024
    • (2024)LiteFlow: Toward High-Performance Adaptive Neural Networks for Kernel DatapathIEEE/ACM Transactions on Networking10.1109/TNET.2023.329315232:1(627-642)Online publication date: Feb-2024
    • (2024)Un-IOV: Achieving Bare-Metal Level I/O Virtualization Performance for Cloud Usage With Migratability, Scalability and TransparencyIEEE Transactions on Computers10.1109/TC.2024.337558973:7(1655-1668)Online publication date: Jul-2024
    • (2024)A Technique for Secure Variant Calling on Human Genome Sequences Using SmartNICs2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00044(328-335)Online publication date: 7-Jul-2024
    • (2024)UniNet: Accelerating the Container Network Data Plane in IaaS Clouds2024 IEEE 17th International Conference on Cloud Computing (CLOUD)10.1109/CLOUD62652.2024.00023(115-127)Online publication date: 7-Jul-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media