Shreya Prasad
ABSTRACT
A typical predictive analytics workflow will pre-process data in a database, transfer the resulting data to an external statistical tool such as R, create machine learning models in R, and then apply the model on newly arriving data. Today, this workflow is slow and cumbersome. Extracting data from databases, using ODBC connectors, can take hours on multi-gigabyte datasets. Building models on single-threaded R does not scale. Finally, it is nearly impossible to use R or other common tools, to apply models on terabytes of newly arriving data.
We solve all the above challenges by integrating HP Vertica with Distributed R, a distributed framework for R. This paper presents the design of a high performance data transfer mechanism, new data-structures in Distributed R to maintain data locality with database table segments, and extensions to Vertica for saving and deploying R models. Our experiments show that data transfers from Vertica are 6x faster than using ODBC connections. Even complex predictive analysis on 100s of gigabytes of database tables can complete in minutes, and is as fast as in-memory systems like Spark running directly on a distributed file system.
- Apache Mahout. http://mahout.apache.org.Google Scholar
- Comparing pattern mining on a billion records with HP Vertica and Hadoop. http://www.vertica.com/2013/04/08/comparing-pattern-mining-on-a-billion-records-with-hp-vertica-and-hadoop/.Google Scholar
- HP Vertica Distributed R. https://www.vertica.com/distributedR.Google Scholar
- MATLAB-parallel computing toolbox. http://www.mathworks.com/products/parallel-computing/.Google Scholar
- MATLAB{the language of technical computing. http://www.mathworks.com/products/matlab.Google Scholar
- Process containers. http://lwn.net/Articles/236038/.Google Scholar
- Programmatic media buying. http://rocketfuel.com/.Google Scholar
- The R project for statistical computing. http://www.r-project.org.Google Scholar
- Revolution R enterprise scaler. http://www.revolutionanalytics.com/revolution-r-enterprise-scaler.Google Scholar
- RHadoop and MapR. https://www.mapr.com/resources/rhadoop-and-mapr.Google Scholar
- SAS analytics. http://www.sas.com.Google Scholar
- SAS in-memory statistics for Hadoop. http://www.sas.com/en_us/software/sas-hadoop/in-memory-hadoop.html.Google Scholar
- Teradata parallel transporter. http://www.teradata.com/tools-and-utilities/parallel-transporter/.Google Scholar
- Teradata Warehouse Miner. http://www.teradata.com/products-and-services/teradata-warehouse-miner/.Google Scholar
- S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. In WWW7, 1998. Google ScholarDigital Library
- Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst. HaLoop: Efficient iterative data processing on large clusters. PVLDB, 3(1-2), 2010. Google ScholarDigital Library
- S. Das, Y. Sismanis, K. S. Beyer, R. Gemulla, P. J. Haas, and J. McPherson. Ricardo: Integrating R and Hadoop. In SIGMOD, 2010. Google ScholarDigital Library
- J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Commun. ACM, 51(1), 2008. Google ScholarDigital Library
- J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, J. Qiu, and G. Fox. Twister: A runtime for iterative MapReduce. In HPDC, 2010. Google ScholarDigital Library
- J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In OSDI, 2012. Google ScholarDigital Library
- P. Gro'e, W. Lehner, T. Weichert, F. Färber, and W.-S. Li. Bridging two worlds with RICE integrating R into the SAP in-memory computing engine. PVLDB, 4(12), 2011.Google Scholar
- J. M. Hellerstein, C. Ré, F. Schoppmann, D. Z. Wang, E. Fratkin, A. Gorajek, K. S. Ng, C. Welton, X. Feng, K. Li, and A. Kumar. The MADlib analytics library: Or MAD skills, the SQL. PVLDB, 5(12), 2012. Google ScholarDigital Library
- A. Lamb, M. Fuller, R. Varadarajan, N. Tran, B. Vandiver, L. Doshi, and C. Bear. The Vertica analytic database: C-store 7 years later. PVLDB, 5(12), 2012. Google ScholarDigital Library
- G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In SIGMOD, 2010. Google ScholarDigital Library
- E. Soroush, M. Balazinska, and D. Wang. ArrayStore: A storage manager for complex parallel array processing. In SIGMOD, 2011. Google ScholarDigital Library
- S. Venkataraman, E. Bodzsar, I. Roy, A. AuYoung, and R. S. Schreiber. Presto: Distributed machine learning and graph processing with sparse matrices. In EuroSys, 2013. Google ScholarDigital Library
- T. J. Ypma. Historical development of the Newton-Raphson method. SIAM Rev., 37(4), 1995. Google ScholarDigital Library
- M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In NSDI, 2012. Google ScholarDigital Library
Index Terms
- Large-scale Predictive Analytics in Vertica: Fast Data Transfer, Distributed Model Creation, and In-database Prediction
Recommendations
The vertica database: SQL RDBMS for managing big data
MBDS '12: Proceedings of the 2012 workshop on Management of big data systemsIn this presentation, we describe the architecture of the Vertica Analytic Database (Vertica), with an emphasis on the management features. Vertica combines a scale-out design, commodity hardware, and the RDBMS data management paradigm to keep SQL ...
Unified analytics platform for big data
WICSA/ECSA '12: Proceedings of the WICSA/ECSA 2012 Companion VolumeGreenplum is using Hadoop and several other open source tools in interesting ways as part of a big data architecture with their Greenplum Database (a scale-out MPP SQL database).
Large-scale multilevel streaming data analytics
CASCON '18: Proceedings of the 28th Annual International Conference on Computer Science and Software EngineeringThere is a monumental shift happening in how data powers organizational and business operations. This shift is about moving away from traditional batch and real-time analytics to hybrid analytics involving both static and continuous data. Most analytics ...
Comments