skip to main content
10.1145/3239235.3240506acmconferencesArticle/Chapter ViewAbstractPublication PagesesemConference Proceedingsconference-collections
research-article

Speeding up mutation testing via the cloud: lessons learned for further optimisations

Published: 11 October 2018 Publication History

Abstract

Background: Mutation testing is the state-of-the-art technique for assessing the fault detection capacity of a test suite. Unfortunately, it is seldom applied in practice because it is computationally expensive. We witnessed 48 hours of mutation testing time on a test suite comprising 272 unit tests and 5,258 lines of test code for testing a project with 48,873 lines of production code. Aims: Therefore, researchers are currently investigating cloud solutions, hoping to achieve sufficient speed-up to allow for a complete mutation test run during the nightly build. Method: In this paper we evaluate mutation testing in the cloud against two industrial projects. Results: With our proof-of-concept, we achieved a speed-up between 12x and 12.7x on a cloud infrastructure with 16 nodes. This allowed to reduce the aforementioned 48 hours of mutation testing time to 3.7 hours. Conclusions: We make a detailed analysis of the delays induced by the distributed architecture, point out avenues for further optimisation and elaborate on the lessons learned for the mutation testing community. Most importantly, we learned that for optimal deployment in a cloud infrastructure, tasks should remain completely independent. Mutant optimisation techniques that violate this principle will benefit less from deploying in the cloud.

References

[1]
Bram Adams and Shane McIntosh. 2016. Modern release engineering in a nutshell-why researchers should care. In Software Analysis, Evolution, and Reengineering (SANER), 2016 IEEE 23rd International Conference on, Vol. 5. IEEE Press, Piscataway, NJ, USA, 78--90.
[2]
Richard Baker and Ibrahim Habli. 2013. An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software. IEEE Transations on Software Engineering 39, 6 (June 2013), 787--805.
[3]
Choi Byoungju and Aditya P Mathur. 1993. High-performance mutation testing. Journal of Systems and Software 20, 2 (1993), 135--152.
[4]
Xia Cai and Michael R Lyu. 2005. The effect of code coverage on fault detection under different testing profiles. ACM SIGSOFT Software Engineering Notes 30, 4 (2005), 1--7.
[5]
Pablo C Cañizares, Mercedes G Merayo, and Alberto Núñez. 2016. EMINENT: EMbarrassINgly parallEl mutatioN Testing. Procedia Computer Science 80 (2016), 63--73.
[6]
Stephen E Deering and David R Cheriton. 1990. Multicast routing in datagram internetworks and extended LANs. ACM Transactions on Computer Systems (TOCS) 8, 2 (1990), 85--110.
[7]
Richard A DeMillo, Edward W Krauser, and Aditya P Mathur. 1991. Compiler-integrated program mutation. In Computer Software and Applications Conference, 1991. COMPSAC'91., Proceedings of the Fifteenth Annual International. IEEE Press, Piscataway, NJ, USA, 351--356.
[8]
Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in Canada. Journal of Systems and Software 86, 5 (2013), 1354--1376.
[9]
G. Gay, M. Staats, M. Whalen, and M. P. E. Heimdahl. 2015. The Risks of Coverage-Directed Test Case Generation. IEEE Transactions on Software Engineering 41, 8 (Aug 2015), 803--819.
[10]
Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th International Conference on Software Engineering. ACM, New York, NY, USA, 72--82.
[11]
Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering. ACM, New York, NY, USA, 435--445.
[12]
Goran Petrović Marko Ivanković, Bob Kurtz, Paul Ammann, and René Just. 2018. An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions. In Proceedings of the International Workshop on Mutation Analysis (Mutation). IEEE Press, Piscataway, NJ, USA, To appear.
[13]
Jon Jenkins. 2011. Velocity Culture. (2011). Keynote Address at the Velocity 2011 Conference.
[14]
Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering 37, 5 (2011), 649--678.
[15]
Susanne Kandl and Sandeep Chandrashekar. 2015. Reasonability of MC/DC for safety-relevant software implemented in programming languages with short-circuit evaluation. Computing 97, 3 (March 2015), 61--279.
[16]
Kim N King and A Jefferson Offutt. 1991. A fortran language system for mutation-based software testing. Software: Practice and Experience 21, 7 (1991), 685--718.
[17]
Philippe Kruchten. 1995. The 4+1 View Model of Architecture. IEEE Software 12, 6 (Nov. 1995), 42--50.
[18]
Rob M. 2014. Everything you need to know about Tesla software updates. (2014). {on line} https://www.teslarati.com/everything-need-to-know-tesla-software-updates/ --- last accessed In May 2018.
[19]
Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-scale Continuous Testing. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP '17). IEEE Press, Piscataway, NJ, USA, 233--242.
[20]
Glenford J Myers. 1979. 77ie Art of Software Testing. (1979).
[21]
SP Ng, Tafline Murnane, Karl Reed, D Grant, and TY Chen. 2004. A preliminary survey on software testing practices in Australia. In Software Engineering Conference, 2004. Proceedings. 2004 Australian. IEEE Press, Piscataway, NJ, USA, 116--125.
[22]
A Jefferson Offutt, Roy P Pargas, Scott V Fichter, and Prashant K Khambekar. 1992. Mutation testing of software using a MIMD computer. In in 1992 International Conference on Parallel Processing. CRC Press, Boca Raton, Florida, II-257--266.
[23]
A Jefferson Offutt and Roland H Untch. 2001. Mutation 2000: Uniting the orthogonal. In Mutation testing for the new century. Springer, Berlin, Germany, 34--44.
[24]
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. Advances in Computers In Press, Corrected Proof (Available online 7 May 2018 2019), ---.
[25]
Ali Parsai, Alessandro Murgia, and Serge Demeyer. 2017. LittleDarwin: a Feature-Rich and Extensible Mutation Testing Framework for Large and Complex Java Systems. In Proceedings FSEN2017 (7th IPM International Conference on Fundamentals of Software Engineering), Mehdi Dastani and Marjan Sirjani (Eds.). Springer International Publishing, Berlin, Germany, 148--163.
[26]
Rudolf Ramler, Thomas Wetzlmaier, and Claus Klammer. 2017. An Empirical Study on the Application of Mutation Testing for a Safety-critical Industrial Software System. In Proceedings of the Symposium on Applied Computing (SAC '17). ACM, New York, NY, USA, 1401--1408.
[27]
Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Engineering 14, 2 (2009), 131--164.
[28]
Iman Saleh and Khaled Nagi. 2015. Hadoopmutator: A cloud-based mutation testing framework. In International Conference on Software Reuse. Springer, Berlin, Germany, 172--187.
[29]
Robert K. Yin. 2002. Case Study Research: Design and Methods, 3 edition. Sage Publications, ---.
[30]
Christian N Zapf. 1993. MedusaMothra-A distributed interpreter for the Mothra mutation testing system. Master's thesis. Clemson University.

Cited By

View all
  • (2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
  • (2023)Parallel mutation testing for large scale systemsCluster Computing10.1007/s10586-023-04074-y27:2(2071-2097)Online publication date: 20-Jun-2023
  • (2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
October 2018
487 pages
ISBN:9781450358231
DOI:10.1145/3239235
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Conference

ESEM '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
  • (2023)Parallel mutation testing for large scale systemsCluster Computing10.1007/s10586-023-04074-y27:2(2071-2097)Online publication date: 20-Jun-2023
  • (2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
  • (2020)Predicting Survived and Killed Mutants2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00053(274-283)Online publication date: Oct-2020
  • (2020)Formal Verification of Developer Tests: A Research Agenda Inspired by Mutation TestingLeveraging Applications of Formal Methods, Verification and Validation: Engineering Principles10.1007/978-3-030-61470-6_2(9-24)Online publication date: 20-Oct-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media