research-article

Priority Based Resource Scheduling Techniques for a Resource Constrained Stream Processing System

Authors:

Rudraneel Chakraborty,

Shikharesh MajumdarAuthors Info & Claims

BDCAT '17: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

Pages 21 - 31

https://doi.org/10.1145/3148055.3148066

Published: 05 December 2017 Publication History

Abstract

A multitenant Storm cluster runs multiple stream processing applications and uses the default Isolation Scheduler to schedule them. Isolation Scheduler assigns resources to topologies based on static resource configuration and does not provide any means for prioritizing topologies based on their varying business requirements. Thus, performance degradation, even complete starvation of topologies with high priority is possible when the cluster is resource constrained and comprises an inadequate number of resources. Two priority based resource scheduling techniques are proposed to overcome these problems. A performance analysis based on prototyping and measurements demonstrates the effectiveness of the proposed techniques.

References

[1]

IBM, "Bringing big data to the enterprise," June 2016. {Online}. Available: https://www-01.ibm.com/software/data/bigdata/what-is-big-data.html. {Accessed July 2016}.

[2]

SAS, "Big Data in Big Companies," June 2016. {Online}. Available: http://www.sas.com/resources/asset/Big-Data-in-Big-Companies-Executive-Summary.pdf. {Accessed July 2016}.

[3]

Gartner, "3D Data Management: Controlling Data Volume, Velocity and Variety," Gartner, November 2016. {Online}. Available: http://blogs.gartner.com/doug-laney/files/2012/01/ad949--3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf. {Accessed 12 September 2016}.

[4]

Apache Hadoop, November 2016. {Online}. Available: http://hadoop.apache.org/. {Accessed September 2016}.

[5]

Apache Spark, June 2016. {Online}. Available: http://spark.apache.org/. {Accessed May 2016}.

[6]

Apache Storm, November 2016. {Online}. Available: http://Storm.apache.org/. {Accessed May 2016}.

[7]

Apache Storm, "Scheduler," {Online}. Available: http://Storm.apache.org/releases/1.0.1/Storm-Scheduler.html.

[8]

Apache Zookeeper, "Apache Zokeeper," June 2016. {Online}. Available: https://zookeeper.apache.org/. {Accessed May 2016}.

[9]

Apache Kafka, "Apache Kafka," June 2016. {Online}. Available: https://kafka.apache.org/. {Accessed May 2016}.

[10]

Oracle, June 2016. {Online}. Available: https://www.oracle.com/java/index.html.

[11]

L. Aniello, R. Baldoni and L. Querzoni, "Adaptive online scheduling in Storm," in International conference on Distributed event-based systems, 2013.

Digital Library

[12]

B. Peng, M. Hosseini, Z. Hong, R. Farivar and R. Campbell, "R-Storm: Resource-Aware Scheduling in Storm," in 16th Annual Middleware Conference (Middleware '15), 2015.

Digital Library

[13]

V. Cardellini, V. Grassi, F. L. Presti and M. Nardelli, "Optimal operator placement for distributed stream processing applications," in 10th ACM International Conference on Distributed and Event-based Systems (DEBS '16), 2016.

Digital Library

[14]

J. Xu, Z. Chen, J. Tang and S. Su, "T-Storm: Traffic-Aware Online Scheduling in Storm," in International Conference on Distributed Computing Systems (ICDCS 13), 2014.

Digital Library

[15]

L. Fischer and A. Bernstein, "Workload scheduling in distributed stream processors using graph partitioning," in International Conference on Big Data (Big Data), 2015.

Digital Library

[16]

Y. Xing, S. Zdonik and J.-H. Hwang, "Dynamic load distribution in the Borealis stream processor," in 21st International Conference on Data Engineering (ICDE '05), April 2005.

Digital Library

[17]

P. Bellavista, A. Corradi, A. Reale and N. Ticca, "Priority-Based Resource Scheduling in Distributed Stream Processing Systems for Big Data Applications," in 7th International Conference on Utility and Cloud Computing (UCC '14), 2014.

Digital Library

[18]

P. Bellavista, A. Corradi and A. Reale, "Design and Implementation of a Scalable and QoS-aware Stream Processing Framework: The Quasit Prototype," in International Conference onGreen Computing and Communications (GreenCom), 2012.

Digital Library

[19]

Apache Storm, April 2016. {Online}. Available: https://Storm.apache.org/releases/0.9.7/javadocs/backtype/Storm/scheduler/IScheduler.html.

[20]

R. Ranjan, "Streaming Big Data Processing in Datacenter Clouds," IEEE Cloud Computing, vol. 1, no. 1, pp. 78--83, 2014.

[21]

N. Marz, Big Data, NY: Manning Publications Co, 2015.

[22]

Intel, "Using real-time data to improve efficiency and reduce total cost of ownership," May 2017. {Online}. Available: http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/real-time-data-improves-efficiency-reduces-ownership-cost-paper.pdf.

[23]

H. Labs, "Innovating for the environment," May 2017. {Online}. Available: http://www.hpl.hp.com/environment/datacenters.html.

[24]

D. C. Luckham, The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, 3 ed., Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2001.

[25]

Twitter, "Handling five billion sessions a day -- in real time," June 2017. {Online}. Available: https://blog.twitter.com/2015/handling-five-billion-sessions-a-day-in-real-time.

[26]

Amazon Web Services. May 2017. {Online}. Available: https://aws.amazon.com/.

[27]

Y. Zhou, B. C. Ooi, K.-L. Tan and J. Wu, "Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System," in Confederated international conference on On the Move to Meaningful Internet Systems: CoopIS, DOA, GADA, and ODBASE - Volume Part I (ODBASE'06/OTM'06), 2006.

Digital Library

[28]

R. Chakraborty, "Priority-Based Scheduling Techniques for a Multitenant Stream Processing Platform", MASc. Thesis, Dept. of Systems and Computer Engineering, Carleton University, Ottawa, Canada, May 2017. {Online}. Available: https://curve.carleton.ca/f27d00de-5403--4e3c-aed2--6b13c08989b1

Cited By

Majumdar S(2024)Streaming Data and Complex Event ProcessingResource Management on Distributed Systems10.1002/9781119912965.ch8(169-200)Online publication date: 6-Sep-2024
https://doi.org/10.1002/9781119912965.ch8
Majumdar S(2018)Leveraging Cloud Computing and Sensor-Based Devices in the Operation and Management of Smart SystemsHandbook of Smart Cities10.1007/978-3-319-97271-8_3(55-80)Online publication date: 16-Nov-2018
https://doi.org/10.1007/978-3-319-97271-8_3

Recommendations

Single machine batch scheduling with deadlines and resource dependent processing times

We consider the problem of scheduling n jobs on a single machine where each job has a deadline and a processing time that is a linear decreasing function of the amount of a common discrete resource allocated to the job. Jobs may be combined to form ...
Resource constrained scheduling on multiple machines

This paper is devoted to RESOURCE CONSTRAINED SCHEDULING. An instance for this problem is given by a set of n jobs with lengths and weights and a set of m machines with capacities. At every time each machine can run arbitrary many jobs in parallel but ...
Machine scheduling with resource dependent processing times

We consider machine scheduling on unrelated parallel machines with the objective to minimize the schedule makespan. We assume that, in addition to its machine dependence, the processing time of any job is dependent on the usage of a discrete renewable ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

BDCAT '17: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies

December 2017

288 pages

ISBN:9781450355490

DOI:10.1145/3148055

General Chairs:
Ashiq Anjum
University of Derby, UK
,
Alan Sill
Texas Tech University, USA
,
Program Chairs:
Xinghui Zhao
Washington State University Vancouver, USA
,
Mohsen Farid
University of Derby, UK
,
Shrideep Pallickara
Colorado State University, USA
,
Jiannong Cao
The HongKong Polytechnic University, Hong Kong

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE TCSC: IEEE Technical Committee on Scalable Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

UCC '17

Sponsor:

SIGARCH
IEEE TCSC

UCC '17: 10th International Conference on Utility and Cloud Computing

December 5 - 8, 2017

Texas, Austin, USA

Acceptance Rates

BDCAT '17 Paper Acceptance Rate 27 of 93 submissions, 29%;

Overall Acceptance Rate 27 of 93 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
94
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Majumdar S(2024)Streaming Data and Complex Event ProcessingResource Management on Distributed Systems10.1002/9781119912965.ch8(169-200)Online publication date: 6-Sep-2024
https://doi.org/10.1002/9781119912965.ch8
Majumdar S(2018)Leveraging Cloud Computing and Sensor-Based Devices in the Operation and Management of Smart SystemsHandbook of Smart Cities10.1007/978-3-319-97271-8_3(55-80)Online publication date: 16-Nov-2018
https://doi.org/10.1007/978-3-319-97271-8_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten