short-paper

The Best of Both Worlds: Big Data Programming with Both Productivity and Performance

Authors:

James ChengAuthors Info & Claims

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

Pages 1619 - 1622

https://doi.org/10.1145/3035918.3058735

Published: 09 May 2017 Publication History

Get Access

Abstract

Coarse-grained operators such as map and reduce have been widely used for large-scale data processing. While they are easy to master, over-simplified APIs sometimes hinder programmers from fine-grained control on how computation is performed and hence designing more efficient algorithms. On the other hand, resorting to domain-specific languages (DSLs) is also not a practical solution, since programmers may need to learn how to use many systems that can be very different from each other, and the use of low-level tools may even result in bug-prone programming.

In [7] our prior work, we proposed Husky which provides a highly expressive API to solve the above dilemma. It allows developers to program in a variety of patterns, such as MapReduce, GAS, vertex-centric programs, and even asynchronous machine learning. While the Husky C++ engine provides great performance, in this demo proposal we introduce PyHusky and ScHusky, which allow users (e.g., data scientists) without system knowledge and low-level programming skills to leverage the performance of Husky and build high-level applications with ease using Python and Scala.

References

[1]

Apache Flink. https://flink.apache.org/.

Google Scholar

[2]

J. Canny and H. Zhao. Butterfly mixing: Accelerating incremental-update algorithms on clusters. In Proceedings of the 13th SIAM International Conference on Data Mining, May 2-4, 2013. Austin, Texas, USA., pages 785--793, 2013.

Google Scholar

[3]

J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In OSDI, pages 17--30, 2012.

Digital Library

Google Scholar

[4]

S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing. On model parallelization and scheduling strategies for distributed machine learning. In NIPS, pages 2834--2842, 2014.

Digital Library

Google Scholar

[5]

J. Li, J. Cheng, Y. Zhao, F. Yang, Y. Huang, H. Chen, and R. Zhao. A comparison of general-purpose distributed systems for data processing. In IEEE International Conference on Big Data, pages 378--383, 2016.

Crossref

Google Scholar

[6]

G. Malewicz, M. H. Austern, A. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large scale graph processing. In SIGMOD, pages 135--146, 2010.

Digital Library

Google Scholar

[7]

F. Yang, J. Li, and J. Cheng. Husky: Towards a more efficient and expressive distributed computing framework. PVLDB, 9(5):420--431, 2016.

Digital Library

Google Scholar

[8]

F. Yang, F. Shang, Y. Huang, J. Cheng, J. Li, Y. Zhao, and R. Zhao. Lftf: A framework for efficient tensor analytics at scale. PVLDB, 10(7), 2017.

Digital Library

Google Scholar

[9]

H. Yun, H. Yu, C. Hsieh, S. V. N. Vishwanathan, and I. S. Dhillon. NOMAD: nonlocking, stochastic multi-machine algorithm for asynchronous and decentralized matrix completion. PVLDB, 7(11):975--986, 2014.

Digital Library

Google Scholar

[10]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, pages 15--28, 2012.

Digital Library

Google Scholar

Cited By

View all

Yang XYuan JYang HKong YZhang HZhao J(2023)A Highly Interactive Honeypot-Based Approach to Network Threat ManagementFuture Internet10.3390/fi1504012715:4(127)Online publication date: 28-Mar-2023
https://doi.org/10.3390/fi15040127
Shaikhha AElseidy MMihaila SEspino DKoch C(2020)Synthesis of Incremental Linear Algebra ProgramsACM Transactions on Database Systems10.1145/338539845:3(1-44)Online publication date: 26-Aug-2020
https://dl.acm.org/doi/10.1145/3385398
Chen HLi CFang JHuang CCheng JZhang JHou YYan X(2019)GrasperProceedings of the ACM Symposium on Cloud Computing10.1145/3357223.3362715(87-100)Online publication date: 20-Nov-2019
https://dl.acm.org/doi/10.1145/3357223.3362715
Show More Cited By

Index Terms

The Best of Both Worlds: Big Data Programming with Both Productivity and Performance
1. Information systems
  1. Data management systems
    1. Database management system engines

Recommendations

Simba: spatial in-memory big data analysis
SIGSPACIAL '16: Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

We present the Simba (Spatial In-Memory Big data Analytics) system, which offers scalable and efficient in-memory spatial query processing and analytics for big spatial data. Simba natively extends the Spark SQL engine to ...
Transformer: A New Paradigm for Building Data-Parallel Programming Models

Cloud computing drives the design and development of diverse programming models for massive data processing. The Transformer programming framework aims to facilitate the building of diverse data-parallel programming models. Transformer has two layers: a ...
Octopus-DF: Unified DataFrame-based cross-platform data analytic system
Abstract
Nowadays, DataFrame serves as a core to model and implement numerous machine learning and data analytic algorithms. Traditional data analytic programming languages, such as Python, provide the DataFrame programming model natively. In ...

Comments

Information & Contributors

Information

Published In

SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

May 2017

1810 pages

ISBN:9781450341974

DOI:10.1145/3035918

General Chairs:
Rada Chirkova
North Carolina State University, USA
,
Jun Yang
Duke University, USA
,
Program Chair:
Dan Suciu
University of Washington, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

ITF
Hong Kong RGC
Research Committee of CUHK

Conference

SIGMOD/PODS'17

Sponsor:

SIGMOD

SIGMOD/PODS'17: International Conference on Management of Data

May 14 - 19, 2017

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
405
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Yang XYuan JYang HKong YZhang HZhao J(2023)A Highly Interactive Honeypot-Based Approach to Network Threat ManagementFuture Internet10.3390/fi1504012715:4(127)Online publication date: 28-Mar-2023
https://doi.org/10.3390/fi15040127
Shaikhha AElseidy MMihaila SEspino DKoch C(2020)Synthesis of Incremental Linear Algebra ProgramsACM Transactions on Database Systems10.1145/338539845:3(1-44)Online publication date: 26-Aug-2020
https://dl.acm.org/doi/10.1145/3385398
Chen HLi CFang JHuang CCheng JZhang JHou YYan X(2019)GrasperProceedings of the ACM Symposium on Cloud Computing10.1145/3357223.3362715(87-100)Online publication date: 20-Nov-2019
https://dl.acm.org/doi/10.1145/3357223.3362715
Chen HLiu MZhao YYan XYan DCheng JOliveira RFelber PHu Y(2018)G-MinerProceedings of the Thirteenth EuroSys Conference10.1145/3190508.3190545(1-12)Online publication date: 23-Apr-2018
https://dl.acm.org/doi/10.1145/3190508.3190545
Huang YJin TWu YCai ZYan XYang FLi JGuo YCheng J(2018)FlexPSProceedings of the VLDB Endowment10.1145/3187009.317773411:5(566-579)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1145/3187009.3177734
Li JYan XZhang JXu ACheng JLiu JNg KCheng TDas GJermaine CBernstein P(2018)A General and Efficient Querying Method for Learning to HashProceedings of the 2018 International Conference on Management of Data10.1145/3183713.3183750(1333-1347)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3183713.3183750
Huang YJin TWu YCai ZYan XYang FLi JGuo YCheng J(2018)FlexPSProceedings of the VLDB Endowment10.1145/3177732.317773411:5(566-579)Online publication date: 5-Oct-2018
https://dl.acm.org/doi/10.1145/3177732.3177734
Zhang QChen HYan DCheng JLoo BBangalore P(2017)Architectural implications on the performance and cost of graph analytics systemsProceedings of the 2017 Symposium on Cloud Computing10.1145/3127479.3128606(40-51)Online publication date: 24-Sep-2017
https://dl.acm.org/doi/10.1145/3127479.3128606
Li JCheng JYang FHuang YZhao YYan XZhao RKando NSakai TJoho HLi Hde Vries AWhite R(2017)LoSHaProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3080800(635-644)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3080800

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Simba: spatial in-memory big data analysis

Transformer: A New Paradigm for Building Data-Parallel Programming Models

Octopus-DF: Unified DataFrame-based cross-platform data analytic system

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations