skip to main content
10.1145/3093742.3095105acmconferencesArticle/Chapter ViewAbstractPublication PagesdebsConference Proceedingsconference-collections
short-paper

Automatic Anomaly Detection over Sliding Windows: Grand Challenge

Published: 08 June 2017 Publication History

Abstract

With the advances in the Internet of Things and rapid generation of vast amounts of data, there is an ever growing need for leveraging and evaluating event-based systems as a basis for building realtime data analytics applications. The ability to detect, analyze, and respond to abnormal patterns of events in a timely manner is as challenging as it is important. For instance, distributed processing environment might affect the required order of events, time-consuming computations might fail to scale, or delays of alarms might lead to unpredicted system behavior. The ACM DEBS Grand Challenge 2017 focuses on real-time anomaly detection for manufacturing equipments based on the observation of a stream of measurements generated by embedded digital and analogue sensors. In this paper, we present our solution to the challenge leveraging the Apache Flink stream processing framework and anomaly ordering based on sliding windows, and evaluate the performance in terms of event latency and throughput.

References

[1]
Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015).
[2]
Margaret H Dunham, Yu Meng, and Jie Huang. 2004. Extensible markov model. In Data Mining, 2004. ICDM'04. Fourth IEEE International Conference on. IEEE, 371--374.
[3]
Raul Castro Fernandez, Matthias Weidlich, Peter Pietzuch, and Avigdor Gal. 2014. Scalable stateful stream processing for smart grids. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems. ACM, 276--281.
[4]
Dani Goldberg and Maja J Matarić. 1999. Coordinating mobile robot group behavior using a model of interaction dynamics. In Proceedings of the third annual conference on Autonomous Agents. ACM, 100--107.
[5]
Vincenzo Gulisano, Zbigniew Jerzak, Roman Katerinenko, Martin Strohbach, and Holger Ziekow. 2017. The DEBS 2017 grand challenge. In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, DEBS '17, Barcelona, Spain, June 19-23, 2017.
[6]
Michael Hahsler, Margaret H Dunham, et al. 2010. remm: Extensible markov model for data stream clustering in r. Journal of Statistical Software 35, 5 (2010), 1--31.
[7]
Apache Jena. 2015. A free and open source Java framework for building Semantic Web and Linked Data applications. Available online: jena. apache. org/(accessed on 28 April 2015) (2015).
[8]
Dirk Merkel. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014, 239 (2014), 2.
[9]
Apache RabbitMQ. 2017. RabbitMQ - Messaging that just works. URL: https://www.rabbitmq.com/ (2017).
[10]
Sebnem Rusitschka and Edward Curry. 2016. Big Data in the Energy and Transport Sectors. In New Horizons for a Data-Driven Economy. Springer, 225--244.
[11]
Omran Saleh, Heiko Betz, and Kai-Uwe Sattler. 2015. Partitioning for scalable complex event processing on data streams. In New Trends in Database and Information Systems II Springer, 185--197.
[12]
Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy, Jignesh M Patel, Sanjeev Kulkarni, Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, et al. 2014. Storm@twitter. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data. ACM, 147--156.
[13]
Nong Ye, Yebin Zhang, and Connie M Borror. 2004. Robustness of the Markov-chain model for cyber-attack detection. IEEE Transactions on Reliability 53, 1 (2004), 116--123.
[14]
Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. (2010).

Cited By

View all
  • (2023)State of the art on quality control for data streamsComputer Science Review10.1016/j.cosrev.2023.10055448:COnline publication date: 1-May-2023
  • (2021)Solving the 2021 DEBS grand challenge using Apache FlinkProceedings of the 15th ACM International Conference on Distributed and Event-based Systems10.1145/3465480.3466929(142-147)Online publication date: 28-Jun-2021
  • (2020)On Making Factories Smarter through Actionable Predictions based on Time-Series DataProcedia Manufacturing10.1016/j.promfg.2020.10.16951(1207-1214)Online publication date: 2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
DEBS '17: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems
June 2017
393 pages
ISBN:9781450350655
DOI:10.1145/3093742
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. K-means
  2. Markov chain model
  3. anomaly detection
  4. event ordering
  5. event-based processing

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Conference

DEBS '17

Acceptance Rates

DEBS '17 Paper Acceptance Rate 22 of 60 submissions, 37%;
Overall Acceptance Rate 145 of 583 submissions, 25%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)State of the art on quality control for data streamsComputer Science Review10.1016/j.cosrev.2023.10055448:COnline publication date: 1-May-2023
  • (2021)Solving the 2021 DEBS grand challenge using Apache FlinkProceedings of the 15th ACM International Conference on Distributed and Event-based Systems10.1145/3465480.3466929(142-147)Online publication date: 28-Jun-2021
  • (2020)On Making Factories Smarter through Actionable Predictions based on Time-Series DataProcedia Manufacturing10.1016/j.promfg.2020.10.16951(1207-1214)Online publication date: 2020
  • (2019)Unveiling Trends and Predictions in Digital Factories2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS)10.1109/DCOSS.2019.00073(326-332)Online publication date: May-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media