skip to main content
10.1145/3214239.3214241acmconferencesArticle/Chapter ViewAbstractPublication PageshpdcConference Proceedingsconference-collections
research-article

Improving Reproducibility of Distributed Computational Experiments

Published: 11 June 2018 Publication History

Abstract

Conference and journal publications increasingly require experiments associated with a submitted article to be repeatable. Authors comply to this requirement by sharing all associated digital artifacts, i.e., code, data, and environment configuration scripts. To ease aggregation of the digital artifacts, several tools have recently emerged that automate the aggregation of digital artifacts by auditing an experiment execution and building a portable container of code, data, and environment. However, current tools only package non-distributed computational experiments. Distributed computational experiments must either be packaged manually or supplemented with sufficient documentation.
In this paper, we outline the reproducibility requirements of distributed experiments using a distributed computational science experiment involving use of message-passing interface (MPI), and propose a general method for auditing and repeating distributed experiments. Using Sciunit we show how this method can be implemented. We validate our method with initial experiments showing application re-execution runtime can be improved by 63% with a trade-off of longer run-time on initial audit execution.

References

[1]
TCPDUMP/LIBPCAP public repository. URL http://www.tcpdump.org/.
[2]
The Sciunit. https://sciunit.run/, 2017. {Online; accessed 10-Sep-2017}.
[3]
Peter Amstutz, Michael R. Crusoe, Neboja Tijanic, Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Herv Mnager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, and Luka Stojanovic. Common Workflow Language, v1.0. 7 2016.
[4]
David H Bailey, Eric Barszcz, John T Barton, David S Browning, Russell L Carter, Leonardo Dagum, Rod A Fatoohi, Paul O Frederickson, Thomas A Lasinski, Rob S Schreiber, et al. The NAS parallel benchmarks. International Journal of High Performance Computing Applications, 5(3):63--73, 1991.
[5]
N. Best, J. Elliott, and I. Foster. Synthesis of a complete land Use/Land cover dataset for the conterminous united states. SSRN eLibrary, 2012.
[6]
Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. ReproZip: Computational reproducibility with ease. In SIGMOD'16, pages 2085--2088, 2016.
[7]
CWL. Common Workflow Language Documentation, 2018. URL https://www.commonwl.org/draft-3/UserGuide.html.
[8]
DePaulDBGroup. Network-enabled Sciunit, 2018. URL https://bitbucket.org/depauldbgroup/provenance-to-use/branch/network.
[9]
William Gropp. MPICH2: a new start for MPI implementations. In Recent Advances in Parallel Virtual Machine and Message Passing Interface, pages 7--7. Springer, 2002.
[10]
Philip J. Guo and Dawson Engler. CDE: Using system call interposition to automatically create portable software packages. In USENIX, 2011.
[11]
Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. The Popper convention: Making reproducible systems evaluation practical. In Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1561--1570. IEEE, 2017.
[12]
Tanu Malik, Ashish Gehani, Dawood Tariq, and Fareed Zaffar. Sketching distributed data provenance. Data Provenance and Data Management in eScience, 426:85--107, 2013.
[13]
Quan Pham, Tanu Malik, and Ian Foster. Using provenance for repeatability. In TaPP, 2013.
[14]
Dai Hai Ton That, Gabriel Fils, Zhihao Yuan, and Tanu Malik. Sciunits: Reusable research objects. In IEEE eScience, Auckland, New Zealand, 2017.
[15]
Aaron Turner and Fred Klassen. TCPreplay - PCAP editing and replaying utilities. URL http://tcpreplay.appneta.com/.
[16]
W3C. PROV-DM: The PROV data model, 2013. URL https://www.w3.org/TR/prov-dm/.
[17]
Tatu Ylonen and Chris Lonvick. The secure shell (SSH) protocol architecture. 2006.
[18]
Zhihao Yuan, Dai Hai Ton That, Siddhant Kothari, Gabriel Fils, and Tanu Malik. Utilizing provenance in reusable research objects. Informatics, 5(1), 2018.

Cited By

View all
  • (2022)Reproducible Notebook Containers using Application Virtualization2022 IEEE 18th International Conference on e-Science (e-Science)10.1109/eScience55777.2022.00015(1-10)Online publication date: Oct-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
P-RECS'18: Proceedings of the First International Workshop on Practical Reproducible Evaluation of Computer Systems
June 2018
37 pages
ISBN:9781450358613
DOI:10.1145/3214239
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Network provenance
  2. Record and replay
  3. Sciunit
  4. reproducibility of distributed objects

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

HPDC '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 22 of 106 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)1
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Reproducible Notebook Containers using Application Virtualization2022 IEEE 18th International Conference on e-Science (e-Science)10.1109/eScience55777.2022.00015(1-10)Online publication date: Oct-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media