skip to main content
10.1145/1298306.1298345acmconferencesArticle/Chapter ViewAbstractPublication PagesimcConference Proceedingsconference-collections
Article

A data streaming algorithm for estimating entropies of od flows

Published: 24 October 2007 Publication History

Abstract

Entropy has recently gained considerable significance as an important metric for network measurement. Previous research has shown its utility in clustering traffic and detecting traffic anomalies. While measuring the entropy of the traffic observed at a single point has already been studied, an interesting open problem is to measure the entropy of the traffic between every origin-destination pair. In this paper, we propose the first solution to this challenging problem. Our sketch builds upon and extends the Lp sketch of Indyk with significant additional innovations. We present calculations showing that our data streaming algorithm is feasible for high link speeds using commodity CPU/memory at a reasonable cost. Our algorithm is shown to be very accurate in practice via simulations, using traffic traces collected at a tier-1 ISP backbone link.

References

[1]
A. Chakrabarti, K. Do Ba, and S. Muthukrishnan. Estimating entropy and entropy norm on data streams. In STACS, 2006.
[2]
L. Bhuvanagiri and S. Ganguly. Estimating entropy over data streams. In ESA, 2006.
[3]
D. Brauckho, B. Tellenbach, A. Wagner, M. May, and A. Lakhina. Impact of packet sampling on anomaly detection metrics. In IMC, 2006.
[4]
G. Casella and R. L. Berger. Statistical Inference. Duxbury, 2nd edition, 2002.
[5]
A. Chakrabarti and G. Cormode. A near-optimal algorithm for computing the entropy of a stream. In SODA, 2007.
[6]
J. M. Chambers, C. L. Mallows, and B. W. Stuck. A method for simulating stable random variables. Journal of the American Statistical Association, 71(354), 1976.
[7]
G. Cormode. Stable distributions for stream computations: It's as easy as 0,1,2. In Workshop on Management and Processing of Data Streams, 2003.
[8]
G. Cormode, P. Indyk, N. Koudas, and S. Muthukrishnan. Fast mining of massive tabular data via approximate distance computations. In ICDE, 2002.
[9]
M. Durand and P. Flajolet. Loglog counting of large cardinalities. In ESA, 2003.
[10]
C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting. In SIGCOMM, Aug. 2002.
[11]
L. Feinstein, D. Schnackenberg, R. Balupari, and D. Kindred. Statistical approaches to DDoS attack detection and response. In Proceedings of the DARPA Information Survivability Conference and Exposition, 2003.
[12]
P. Indyk. Stable distributions, pseudorandom generators, embeddings and data stream computation. In FOCS, 2000.
[13]
P. Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3):307--323, 2006.
[14]
A. Kuzmanovic and E. W. Knightly. Low-rate tcp targeted denial of service attacks (the shrew vs. the mice and elephants). In SIGCOMM, 2003.
[15]
A. Lakhina, M. Crovella, and C. Diot. Mining anomalies using traffic feature distributions. In SIGCOMM, 2005.
[16]
A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In SIGMETRICS, 2006.
[17]
G. S. Manku and R. Motwani. Approximate frequency counts over data streams. In Proceedings of the 28th International Conference on Very Large Data Bases, 2002.
[18]
A. Medina, N. Taft, K. Salamatian, S. Bhattacharyya, and C. Diot. Traffic matrix estimation: existing techniques and new directions. In SIGCOMM, Aug. 2002.
[19]
S. Muthukrishnan. Data streams: algorithms and applications. available athttp://athos.rutgers.edu/~muthu/.
[20]
J. Nolan. STABLE program. online at http://academic2.american.edu/~jpnolan/stable/stable.html.
[21]
A. Soule, A. Nucci, R. Cruz, E. Leonardi, and N. Taft. How to identify and estimate the largest traffic matrix elements in a dynamic environment. In SIGMETRICS, June 2004.
[22]
C. Tebaldi and M. West. Bayesian inference on network traffic using link count data. Journal of American Statistics Association, pages 557--576, 1998.
[23]
Y. Vardi. Internet tomography: estimating source-destination traffic intensities from link data. Journal of American Statistics Association, pages 365--377, 1996.
[24]
A. Wagner and B. Plattner. Entropy Based Worm and Anomaly Detection in Fast IP Networks. In Proceedings of IEEE International Workshop on Enabling Technologies, Infrastructures for Collaborative Enterprises, 2005.
[25]
K. Xu, Z.-L. Zhang, and S. Bhattacharya. Proling internet backbone traffic: Behavior models and applications. In SIGCOMM, 2005.
[26]
Y. Zhang, Z. M. Mao, and J. Wang. Low-rate tcp-targeted dos attack disrupts internet routing. In Proc. 14th Annual Network & Distributed System Security Symposium, 2007.
[27]
Q. Zhao, Z. Ge, J. Wang, and J. Xu. Robust traffic matrix estimation with imperfect information: making use of multiple data sources. In SIGMETRICS, 2006.
[28]
Q. Zhao, A. Kumar, J. Wang, and J. Xu. Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices. In SIGMETRICS, June 2005.
[29]
V. M. Zolotarev. One-Dimensional Stable Distributions, volume 65 of Translations of Mathematical Monographs. American Mathematical Society, Providence, RI, 1986.

Cited By

View all
  • (2024)DynATOS+: A Network Telemetry System for Dynamic Traffic and Query WorkloadsIEEE/ACM Transactions on Networking10.1109/TNET.2024.336743232:4(2810-2825)Online publication date: Aug-2024
  • (2023)Secure federated correlation test and entropy estimationProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619532(26990-27010)Online publication date: 23-Jul-2023
  • (2022)Sketch-based entropy estimationProceedings of the 5th International Workshop on P4 in Europe10.1145/3565475.3569082(57-60)Online publication date: 9-Dec-2022
  • Show More Cited By

Index Terms

  1. A data streaming algorithm for estimating entropies of od flows

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    IMC '07: Proceedings of the 7th ACM SIGCOMM conference on Internet measurement
    October 2007
    390 pages
    ISBN:9781595939081
    DOI:10.1145/1298306
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 October 2007

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. data streaming
    2. entropy estimation
    3. network measurement
    4. stable distributions
    5. traffic matrix

    Qualifiers

    • Article

    Conference

    IMC07
    Sponsor:
    IMC07: Internet Measurement Conference
    October 24 - 26, 2007
    California, San Diego, USA

    Acceptance Rates

    Overall Acceptance Rate 277 of 1,083 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)11
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)DynATOS+: A Network Telemetry System for Dynamic Traffic and Query WorkloadsIEEE/ACM Transactions on Networking10.1109/TNET.2024.336743232:4(2810-2825)Online publication date: Aug-2024
    • (2023)Secure federated correlation test and entropy estimationProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619532(26990-27010)Online publication date: 23-Jul-2023
    • (2022)Sketch-based entropy estimationProceedings of the 5th International Workshop on P4 in Europe10.1145/3565475.3569082(57-60)Online publication date: 9-Dec-2022
    • (2022)Approximately Counting Butterflies in Large Bipartite Graph StreamsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.306298734:12(5621-5635)Online publication date: 1-Dec-2022
    • (2022)Tabular Interpolation Approach Based on Stable Random Projection for Estimating Empirical Entropy of High-Speed Network TrafficIEEE Access10.1109/ACCESS.2022.321033610(104934-104953)Online publication date: 2022
    • (2020)Practical Range Counting over Data Streams2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378146(659-668)Online publication date: 10-Dec-2020
    • (2019)Continuously Distinct Sampling over Centralized and Distributed High Speed Data StreamsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.286545230:2(300-314)Online publication date: 1-Feb-2019
    • (2019)HeavyKeeperIEEE/ACM Transactions on Networking10.1109/TNET.2019.293386827:5(1845-1858)Online publication date: 1-Oct-2019
    • (2019)Detecting a Variety of Long-Term Stealthy User Behaviors on High Speed LinksIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2018.287331931:10(1912-1925)Online publication date: 1-Oct-2019
    • (2019)Utilizing Dynamic Properties of Sharing Bits and Registers to Estimate User Cardinalities Over Time2019 IEEE 35th International Conference on Data Engineering (ICDE)10.1109/ICDE.2019.00101(1094-1105)Online publication date: Apr-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media