Event dissemination via group-aware stream filtering

ABSTRACT

We consider a distributed system that disseminates high-volume event streams to many simultaneous monitoring applications over a low-bandwidth network. For bandwidth efficiency, we propose a group-aware stream filtering approach, used together with multicasting, that exploits two overlooked, yet important, properties of monitoring applications: 1) many of them can tolerate some degree of "slack" in their data quality requirements, and 2) there may exist multiple subsets of the source data satisfying the quality needs of an application. We can thus choose the "best alternative" subset for each application to maximize the data overlap within the group to best benefit from multicasting. We provide a general framework that treats the group-aware stream filtering problem completely; we prove the problem NP-hard and thus provide a suite of heuristic algorithms that ensure data quality (specifically, granularity and timeliness) while preserving bandwidth. Our evaluation shows that group-aware stream filtering is effective in trading CPU time for bandwidth savings, compared with self-interested filtering.

REFERENCES

• 1: S. Aryangat, H. Andrade, and A. Sussman. Time and space optimization for processing groups of multi-dimensional scientific queries. In Proceedings of the 18th Annual International Conference on Supercomputing (ICS), pages 95--105, 2004.
• 2: B. Babcock, M. Datar, and R. Motwani. Sampling from a moving window over streaming data. In Proceedings of the Thirteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 633--634, 2002.
• 3: Z. Bar-Yossef, R. Kumar, and D. Sivakumar. Sampling algorithms: lower bounds and applications. In Proceedings of the Thirty-third Annual ACM Symposium on Theory of Computing (STOC), pages 266--275, 2001.
• 4: S. Chaudhuri, R. Motwani, and V. Narasayya. On random sampling over joins. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 263--274, 1999.
• 5: G. Chen, M. Li, and D. Kotz. Design and implementation of a large-scale context fusion network. In Proceedings of the First Annual International Conference on Mobile and Ubiquitous Systems (MobiQuitous), pages 246--255. ACM Press, 2004.
• 6: J. Chen, D. J. DeWitt, F. Tian, and Y. Wang. NiagaraCQ: a scalable continuous query system for Internet databases. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 379--390, 2000.
• 7: R. Cheng, B. Kao, S. Prabhakar, A. Kwan, and Y. Tu. Adaptive stream filters for entity-based queries with non-value tolerance. In Proceedings of the 31st International Conference on Very Large Data Bases (VLDB), pages 37--48, 2005.
• 8: T. Cormen, C. Leiserson, R. Rivest, and C. Stein. Introduction to Algorithms. MIT Press, second edition, 2001.
• 9: T. Johnson, S. Muthukrishnan, and I. Rozenbaum. Sampling algorithms in a stream operator. In Proceedings of the 2005 ACM SIGMOD international conference on Management of data (SIGMOD), pages 1--12. ACM Press, 2005.
• 10: M. Li. Group-Aware Stream Filtering. PhD thesis, Dartmouth College Computer Science, Hanover, NH, May 2008. Available as Technical Report TR2008-621.
• 11: S. Madden, M. Shah, J. M. Hellerstein, and V. Raman. Continuously adaptive continuous queries over streams. In Proceedings of the 2002 ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 49--60. ACM Press, 2002.
• 12: D. P. Mitchell. Consequences of stratified sampling in graphics. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 277--280. ACM Press, 1996.
• 13: C. Olston, J. Jiang, and J. Widom. Adaptive filters for continuous queries over distributed data streams. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data (SIGMOD), pages 563--574, 2003.
• 14: R. Strom, G. Banavar, T. Chandra, M. Kaplan, K. Miller, B. Mukherjee, D. Sturman, and M. Ward. Gryphon: An information flow based approach to message brokering. In International Symposium on Software Reliability Engineering (ISSRE), 1998.
• 15: Y. Zhao and R. Strom. Exploiting event stream interpretation in publish-subscribe systems. In Proceedings of the 20th Annual ACM Symposium on Principles of Distributed Computing (PODC), pages 219--228, 2001.