research-article

Typhoon: An SDN Enhanced Real-Time Big Data Streaming Framework

Authors:

Hyunseok Chang,

Sarit Mukherjee,

T. V. Lakshman,

Jacobus Van der MerweAuthors Info & Claims

CoNEXT '17: Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies

Pages 310 - 322

https://doi.org/10.1145/3143361.3143398

Published: 28 November 2017 Publication History

Abstract

Stream processing pipelines operated by current big data streaming frameworks present two problems. First, the pipelines are not flexible, controllable, and programmable enough to accommodate dynamic streaming application needs. Second, the application-level data routing over the pipelines do not exhibit optimal performance for increasingly common one-to-many communication. To address these problems, we propose an SDN-based real-time big data streaming framework called Typhoon, that tightly integrates SDN functionality into a real-time stream framework. By partially offloading application-layer data routing and control to the network layer via SDN interfaces and protocols, Typhoon provides on-the-fly programmability of both the application and network layers, and achieve high-performance data routing. In addition, Typhoon SDN controller exposes cross-layer information, from both the application and the network, to SDN control plane applications to extend the framework's functionality. We introduce several SDN control plane applications to illustrate these benefits.

References

[1]

Apache Apex. https://apex.apache.org.

[2]

Apache Curator. http://curator.apache.org.

[3]

Apache Flink. https://flink.apache.org.

[4]

Apache Gearpump. http://gearpump.apache.org.

[5]

Apache Samza. http://samza.apache.org.

[6]

Apache Spark. http://spark.apache.org.

[7]

Apache Storm. http://storm.apache.org.

[8]

Apex Application Developer Guide. https://github.com/DataTorrent/docs/blob/master/docs/application_development.md.

[9]

Common Topology Patterns. http://storm.apache.org/releases/1.0.3/Common-patterns.html.

[10]

Data Plane Development Kit. http://dpdk.org.

[11]

Gartner Says the Internet of Things Installed Base Will Grow to 26 Billion Units By 2020. https://www.gartner.com/newsroom/id/2636073.

[12]

Gearpump - dynamic DAG. http://mail-archives.apache.org/mod_mbox/incubator-gearpump-user/201609.mbox/browser. Gearpump User Mailing List.

[13]

How Spotify Scales Apache Storm. https://labs.spotify.com/2015/01/05/how-spotify-scales-apache-storm/.

[14]

Microsft Azure SLA for Stream Analytics. https://azure.microsoft.com/support/legal/sla/stream-analytics/.

[15]

New Tweets per second record, and how! https://blog.twitter.com/engineering/en_us/a/2013/new-tweets-per-second-record-and-how.html.

[16]

Project Floodlight. http://www.projectfloodlight.org/floodlight/.

[17]

STORM-634: Storm serialization changed to thrift to support rolling upgrade. https://github.com/apache/storm/pull/414.

[18]

Storm Guaranteeing Message Processing. http://storm.apache.org/releases/current/Guaranteeing-message-processing.html.

[19]

Storm topology event inspector. http://storm.apache.org/releases/2.0.0-SNAPSHOT/Eventlogging.html.

[20]

Stream groupings. http://storm.apache.org/releases/2.0.0-SNAPSHOT/Concepts.html.

[21]

Support for Dynamic Topology. http://mail-archives.apache.org/mod_mbox/apex-users/201608.mbox/browser. Apex Users Mailing List.

[22]

Tigon. http://tigon.io.

[23]

Understanding the Parallelism of a Storm Topology. http://storm.apache.org/releases/2.0.0-SNAPSHOT/Understanding-the-parallelism-of-a-Storm-topology.html.

[24]

OpenFlow Switch Specification. https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-switch-v1.5.0.noipr.pdf, 2014.

[25]

D. J. Abadi, Y. Ahmad, M. Balazinska, U. Cetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, et al. The Design of the Borealis Stream Processing Engine. In Proc. CIDR, 2005.

[26]

A. Al-Shabibi et al. OpenVirteX: Make Your Virtual SDNs Programmable. In Proc. HotSDN, 2014.

Digital Library

[27]

A. Botta, W. de Donato, V. Persico, and A. Pescapè. On the Integration of Cloud Computing and Internet of Things. In Proc. IEEE International Conference on Future Internet of Things and Cloud, 2014.

Digital Library

[28]

Z. Bozakov and P. Papadimitriou. AutoSlice: Automated and Scalable Slicing for Software-Defined Networks. In Proc. ACM CoNEXT, 2012.

Digital Library

[29]

M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. Cetintemel, Y. Xing, and S. B. Zdonik. Scalable Distributed Stream Processing. In Proc. CIDR, 2003.

[30]

S. Chintapalli et al. Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming. In Proc. IEEE International Parallel and Distributed Processing Symposium Workshops, 2016.

[31]

L. G. D. Estrin and M. S. G. Pottie. Instrumenting the World with Wireless Sensor Networks. In Proc. IEEE ICASSP, 2001.

[32]

R. Evans. From Gust to Tempest: Scaling Storm. In Proc. Hadoop Summit, 2015.

[33]

A. D. Ferguson, A. Guha, C. Liang, R. Fonseca, and S. Krishnamurthi. Participatory Networking: An API for Application Control of SDNs. In Proc. ACM SIGCOMM, 2013.

Digital Library

[34]

A. Floratou, A. Agrawal, B. Graham, S. Rao, and K. Ramasamy. Dhalion: Self-Regulating Stream Processing in Heron. VLDB Endowment, 10(12), 2017.

Digital Library

[35]

D. Ford, F. Labelle, F. I. Popovici, M. Stokely, and V.-A. Truong. Availability in Globally Distributed Storage Systems. In Proc. USENIX OSDI, 2010.

Digital Library

[36]

A. Ghoting and S. Parthasarathy. Facilitating Interactive Distributed Data Stream Processing and Mining. In Proc. IEEE International Parallel and Distributed Processing Symposium, 2004.

[37]

S. Hendrickson, S. Sturdevant, T. Harter, V. Venkataramani, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Serverless Computation with OpenLambda. In Proc. 8th USENIX Workshop on HotCloud, 2016.

Digital Library

[38]

P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. ZooKeeper: Wait-free Coordination for Internet-scale Systems. In Proc. USENIX ATC, 2010.

Digital Library

[39]

E. Jonas, Q. Pu, S. Venkataraman, I. Stoica, and B. Recht. Occupy the Cloud: Distributed Computing for the 99%. In Proc. ACM Symposium on Cloud Computing, 2017.

Digital Library

[40]

S. Kamburugamuve, S. Ekanayake, M. Pathirage, and G. Fox. Towards High Performance Processing of Streaming Data in Large Data Centers. In Proc. IEEE International Parallel and Distributed Processing Symposium Workshops, 2016.

[41]

V. Kawadia and P. Kumar. A Cautionary Perspective on Cross-layer Design. IEEE Wireless Communications, 12(1), 2005.

Digital Library

[42]

A. Khrabrov and E. de Lara. Accelerating Complex Data Transfer for Cluster Computing. In Proc. 8th USENIX Workshop on HotCloud, 2016.

Digital Library

[43]

S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J. M. Patel, K. Ramasamy, and S. Taneja. Twitter Heron: Stream Processing at Scale. In Proc. ACM SIGMOD, 2015.

Digital Library

[44]

O. Michel, M. Coughlin, and E. Keller. Extending the Software-Defined Network Boundary. ACM SIGCOMM Computer Communication Review, 44(4):381--382, 2014.

Digital Library

[45]

R. Moore. A Universal Dynamic Trace for Linux and other Operating Systems. In Proc. USENIX ATC, FREENIX Track, 2001.

Digital Library

[46]

M. A. U. Nasir. Fault Tolerance for Stream Processing Engines. arXiv preprint arXiv:1605.00928, 2016.

[47]

M. A. U. Nasir, G. D. F. Morales, D. García-Soriano, N. Kourtellis, and M. Serafini. The Power of Both Choices: Practical Load Balancing for Distributed Stream Processing Engines. In Proc. IEEE International Conference on Data Engineering, 2015.

[48]

L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed Stream Computing Platform. In Proc. IEEE International Conference on Data Mining Workshops, 2010.

Digital Library

[49]

Z. Qian et al. TimeStream: Reliable Stream Computation in the Cloud. In Proc. EuroSys, 2013.

Digital Library

[50]

R. K. Sahoo, A. Sivasubramaniam, M. S. Squillante, and Y. Zhang. Failure Data Analysis of a Large-Scale Heterogeneous Server Environment. In Proc. IEEE International Conference on Dependable Systems and Networks, 2004.

Digital Library

[51]

M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin. Flux: An Adaptive Partitioning Operator for Continuous Query Systems. In Proc. IEEE International Conference on Data Engineering, 2003.

[52]

R. Sherwood et al. FlowVisor: A Network Virtualization Layer. In OpenFlow Switch Consortium, 2009.

[53]

M. Slee, A. Agarwal, and M. Kwiatkowski. Thrift: Scalable Cross-Language Services Implementation. Facebook White Paper, 5(8), 2007.

[54]

M. Stonebraker, U. Çetintemel, and S. Zdonik. The 8 Requirements of Real-Time Stream Processing. ACM SIGMOD Record, 34(4), 2005.

Digital Library

[55]

A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, et al. Storm @Twitter. In Proc. ACM SIGMOD, 2014.

Digital Library

[56]

L. Wang. Usages and Optimizations of Spark at Tencent. In Proc. Big Data Conference, 2015.

[57]

B. White et al. An Integrated Experimental Environment for Distributed Systems and Networks. ACM SIGOPS Operating Systems Review, 36(SI), 2002.

Digital Library

[58]

Y. Xing, S. Zdonik, and J.-H. Hwang. Dynamic load distribution in the borealis stream processor. In Proc. IEEE International Conference on Data Engineering, 2005.

Digital Library

[59]

H. Zhang, G. Ananthanarayanan, P. Bodik, M. Philipose, P. Bahl, and M. J. Freedman. Live Video Analytics at Scale with Approximation and Delay-Tolerance. In Proc. USENIX NSDI, 2017.

Digital Library

[60]

Q. Zhao, R. Rabbah, S. Amarasinghe, L. Rudolph, and W.-F. Wong. How to Do a Million Watchpoints: Efficient Debugging Using Dynamic Instrumentation. In Proc. International Conference on Compiler Construction, 2008.

Digital Library

Cited By

Boughzala BKoldehofe B(2024)In-Network Management of Parallel Data Streams over Programmable Data Planes2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619812(50-58)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619812
Wijesekara PGunawardena S(2023)A Comprehensive Survey on Knowledge-Defined NetworkingTelecom10.3390/telecom40300254:3(477-596)Online publication date: 2-Aug-2023
https://doi.org/10.3390/telecom4030025
Chang HHao FKodialam MLakshman TMukherjee SVarvello M(2023)Towards network-assisted publish–subscribe over wide area networksComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109702231:COnline publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109702
Show More Cited By

Index Terms

Typhoon: An SDN Enhanced Real-Time Big Data Streaming Framework
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
2. Networks
  1. Network protocols
    1. Cross-layer protocols
  2. Network services
    1. Programmable networks

Recommendations

Network Assisted Content Distribution for Adaptive Bitrate Video Streaming
MMSys'17: Proceedings of the 8th ACM on Multimedia Systems Conference

State-of-the-art Software Defined Wide Area Networks (SD-WANs) provide the foundation for flexible and highly resilient networking. In this work we design, implement and evaluate a novel architecture (denoted SABR) that leverages the benefits of SDN to ...
SABR: Network-Assisted Content Distribution for QoE-Driven ABR Video Streaming

State-of-the-art software-defined wide area networks (SD-WANs) provide the foundation for flexible and highly resilient networking. In this work, we design, implement, and evaluate a novel architecture (denoted as SABR) that leverages the benefits of ...
Time-sensitive software-defined networking (Ts-SDN) control architecture for flexi-grid optical networks with data center application

Data center interconnected by flexi-grid optical networks is a promising scenario to meet the high burstiness and high bandwidth requirement of data center application, because flexi-grid optical networks can allocate spectral resources for applications ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CoNEXT '17: Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies

November 2017

492 pages

ISBN:9781450354226

DOI:10.1145/3143361

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CoNEXT '17

Sponsor:

SIGCOMM

CoNEXT '17: The 13th International Conference on emerging Networking EXperiments and Technologies

December 12 - 15, 2017

Incheon, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 198 of 789 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
534
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Boughzala BKoldehofe B(2024)In-Network Management of Parallel Data Streams over Programmable Data Planes2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619812(50-58)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619812
Wijesekara PGunawardena S(2023)A Comprehensive Survey on Knowledge-Defined NetworkingTelecom10.3390/telecom40300254:3(477-596)Online publication date: 2-Aug-2023
https://doi.org/10.3390/telecom4030025
Chang HHao FKodialam MLakshman TMukherjee SVarvello M(2023)Towards network-assisted publish–subscribe over wide area networksComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109702231:COnline publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109702
Chung JZacherek WWisniewski ALiu ZBicer TKettimuthu RFoster IWeissman JChandra AGavrilovska ATiwari D(2022)SciStreamProceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing10.1145/3502181.3531475(185-198)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3502181.3531475
Li WLiu DChen KLi KQi H(2021) Hone: Mitigating Stragglers in Distributed Stream Processing With Tuple Scheduling IEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.305105932:8(2021-2034)Online publication date: 1-Aug-2021
https://doi.org/10.1109/TPDS.2021.3051059
Xing JTian JJiang ZCheng JYin H(2021)Jupiter: a modern federated learning platform for regional medical careScience China Information Sciences10.1007/s11432-020-3062-864:10Online publication date: 9-Sep-2021
https://doi.org/10.1007/s11432-020-3062-8
Wang WTammana PChen ANg T(2020)Grasp the Root Causes in the Data PlaneProceedings of the Symposium on SDN Research10.1145/3373360.3380835(55-61)Online publication date: 3-Mar-2020
https://dl.acm.org/doi/10.1145/3373360.3380835
Fais AGiordano SProcissi G(2020)On the Design of Fast and Scalable Network Applications Through Data Stream Processing2020 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN)10.1109/NFV-SDN50289.2020.9289855(150-154)Online publication date: 10-Nov-2020
https://doi.org/10.1109/NFV-SDN50289.2020.9289855
Chen P(2020)Review of Ubiquitous Computing Based on Markov Chain and High Complexity Data Streaming Model2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT)10.1109/ICSSIT48917.2020.9214168(720-723)Online publication date: Aug-2020
https://doi.org/10.1109/ICSSIT48917.2020.9214168
Barijough KZhao ZGerstlauer A(2019)Quality/Latency-Aware Real-time Scheduling of Distributed Streaming IoT ApplicationsACM Transactions on Embedded Computing Systems10.1145/335820918:5s(1-23)Online publication date: 8-Oct-2019
https://dl.acm.org/doi/10.1145/3358209
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten