skip to main content
10.1145/1247480.1247535acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
Article

Sharing aggregate computation for distributed queries

Published: 11 June 2007 Publication History

Abstract

An emerging challenge in modern distributed querying is to efficiently process multiple continuous aggregation queries simultaneously. Processing each query independently may be infeasible, so multi-query optimizations are critical for sharing work across queries. The challenge is to identify overlapping computations that may not be obvious in the queries themselves.
In this paper, we reveal new opportunities for sharing work in the context of distributed aggregation queries that vary in their selection predicates. We identify settings in which a large set of q such queries can be answered by executing k << q different queries. The k queries are revealed by analyzing a boolean matrix capturing the connection between data and the queries that they satisfy, in a manner akin to familiar techniques like Gaussian elimination. Indeed, we identify a class of linear aggregate functions (including SUM, COUNT and AVERAGE), and show that the sharing potential for such queries can be optimally recovered using standard matrix decompositions from computational linear algebra. For some other typical aggregation functions (including MIN and MAX) we find that optimal sharing maps to the NP-hard set basis problem. However, for those scenarios, we present a family of heuristic algorithms and demonstrate that they perform well for moderate-sized matrices. We also present a dynamic distributed system architecture to exploit sharing opportunities, and experimentally evaluate the benefits of our techniques via a novel, flexible random workload generator we develop for this setting.

References

[1]
R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. M. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. V. der Vorst. Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods. SIAM, 1994.
[2]
S. Cohen and Y. Matias. Spectral bloom filters. In SIGMOD, 2003.
[3]
G. Cormode and M. Garofalakis. "Sketching Streams Through the Net: Distributed Approximate Query Tracking". In VLDB, 2005.
[4]
G. Cormode, M. Garofalakis, S. Muthukrishnan, and R. Rastogi. "Holistic Aggregates in a Networked World: Distributed Tracking of Approximate Quantiles". In SIGMOD, 2005.
[5]
C. Cranor, T. Johnson, O. Spatscheck, and V. Shkapenyuk. "Gigascope: A Stream Database for Network Applications". In SIGMOD, 2003.
[6]
S. Ganguly, M. Garofalakis, and R. Rastogi. Processing set expressions over continuous update streams. In SIGMOD, 2003.
[7]
M. Garey and D. Johnson. "Computers and Intractability: A Guide to the Theory of NP-Completeness". W.H. Freeman, 1979.
[8]
M. B. Greenwald and S. Khanna. "Power-Conserving Computation of Order-Statistics over Sensor Networks". In PODS, 2004.
[9]
R. Huebsch, B. N. Chun, J. M. Hellerstein, B. T. Loo, P. Maniatis, T. Roscoe, S. Shenker, I. Stoica, and A. R. Yumerefendi. "The Architecture of PIER: An Internet-Scale Query Processor". In CIDR, 2005.
[10]
S. Krishnamurthy, C. Wu, and M. Franklin. On-the-fly sharing for streamed aggregation. In SIGMOD, 2006.
[11]
J. Li, D. Maier, K. Tufte, V. Papadimos, and P. A. Tucker. No pane, no gain: efficient evaluation of sliding-window aggregates over data streams. SIGMOD Rec., 34(1), 2005.
[12]
A. Lubiw. The boolean basis problem and how to cover some polygons by rectangles. SIAM Journal on Discrete Mathematics, 3(1), 1990.
[13]
C. Lund and N. Yannakakis. "On the Hardness of Approximating Minimization Problems". Journal of the ACM, 41(5), 1994.
[14]
S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. TAG: A Tiny AGgregation Service for Ad-Hoc Sensor Networks. In OSDI, 2002.
[15]
S. Madden, M. Shah, J. M. Hellerstein, and V. Raman. Continuously Adaptive Continuous Queries. In SIGMOD, 2002.
[16]
S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. "The Design of an Acquisitional Query Processor for Sensor Networks". In SIGMOD, 2003.
[17]
T. K. Sellis. "Multiple Query Optimization". In TODS, 1988.
[18]
L. J. Stockmeyer. The minimal set basis problem in NP-complete. Tech. Report RC 5431, IBM Research, May 1975.
[19]
N. Trigoni, Y. Yao, A. J. Demers, J. Gehrke, and R. Rajaraman. Multi-query optimization for sensor networks. In DCOSS, 2005.
[20]
P. Yalagandula and M. Dahlin. SDIMS: A Scalable Distributed Information Management System. In SIGCOMM, Portland, Oregon, 2004.
[21]
R. Zhang, N. Koudas, B. C. Ooi, and D. Srivastava. Multiple aggregations over data streams. In SIGMOD, 2005.

Cited By

View all
  • (2024)Sharing Queries with Nonequivalent User-defined Aggregate FunctionsACM Transactions on Database Systems10.1145/364913349:2(1-46)Online publication date: 10-Apr-2024
  • (2021)ScottyACM Transactions on Database Systems10.1145/343367546:1(1-46)Online publication date: 27-Mar-2021
  • (2020)Shared Execution Techniques for Business Data Analytics over Big Data StreamsProceedings of the 32nd International Conference on Scientific and Statistical Database Management10.1145/3400903.3400932(1-4)Online publication date: 7-Jul-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of data
June 2007
1210 pages
ISBN:9781595936868
DOI:10.1145/1247480
  • General Chairs:
  • Lizhu Zhou,
  • Tok Wang Ling,
  • Program Chair:
  • Beng Chin Ooi
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2007

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. aggregation
  2. duplicate insensitive
  3. linear algebra
  4. multi-query optimization

Qualifiers

  • Article

Conference

SIGMOD/PODS07
Sponsor:

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)1
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Sharing Queries with Nonequivalent User-defined Aggregate FunctionsACM Transactions on Database Systems10.1145/364913349:2(1-46)Online publication date: 10-Apr-2024
  • (2021)ScottyACM Transactions on Database Systems10.1145/343367546:1(1-46)Online publication date: 27-Mar-2021
  • (2020)Shared Execution Techniques for Business Data Analytics over Big Data StreamsProceedings of the 32nd International Conference on Scientific and Statistical Database Management10.1145/3400903.3400932(1-4)Online publication date: 7-Jul-2020
  • (2018)A Real-Time Sensor Network Aggregation Computing System2018 5th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2018 4th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom)10.1109/CSCloud/EdgeCom.2018.00016(35-40)Online publication date: Jun-2018
  • (2017)One for All and All for OneProceedings of the 11th ACM International Conference on Distributed and Event-based Systems10.1145/3093742.3093918(203-214)Online publication date: 8-Jun-2017
  • (2016)CASQDProceedings of the 10th ACM International Conference on Distributed and Event-based Systems10.1145/2933267.2933316(226-237)Online publication date: 13-Jun-2016
  • (2016)Frugal topology construction for stream aggregation in the cloudIEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications10.1109/INFOCOM.2016.7524534(1-9)Online publication date: Apr-2016
  • (2016)An Efficient Approach of Processing Multiple Continuous QueriesJournal of Computer Science and Technology10.1007/s11390-016-1693-831:6(1212-1227)Online publication date: 9-Nov-2016
  • (2016)Towards Neighborhood Window Analytics over Large-Scale GraphsProceedings, Part II, of the 21st International Conference on Database Systems for Advanced Applications - Volume 964310.1007/978-3-319-32049-6_13(201-217)Online publication date: 16-Apr-2016
  • (2015)Minimizing the Communication Cost of Aggregation in Publish/Subscribe Systems2015 IEEE 35th International Conference on Distributed Computing Systems10.1109/ICDCS.2015.54(462-473)Online publication date: Jun-2015
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media