Diego Rodríguez-Baquero
ABSTRACT
Mutation testing is a technique in which faults (mutants) are injected into a program or application to assess its test suite effectiveness. It works by inserting mutants and running the application’s test suite to identify if the mutants are detected (killed) or not (survived) by the tests. Although computationally expensive, it has proven to be an effective method to assess application test suites. Several mutation testing frameworks and tools have been built for the various programing languages, however, very few tools have been built for the JavaScript language, more specifically, there is a lack of mutation testing tools for the Node.js runtime and npm based applications. The npm Registry is a public collection of modules of open-source code for Node.js, front-end web applications, mobile applications, robots, routers, and countless other needs of the JavaScript community. The over 700,000 packages hosted in npm are downloaded more than 5 billion times per week. More and more software is published in npm every day, representing a huge opportunity to share code and solutions, but also to share bugs and faulty software. In this paper, we briefly describe prior work for mutation operators in JavaScript and Node.js, and propose Mutode, an open source tool which leverages the npm package ecosystem to perform mutation testing for JavaScript and Node.js applications. We empirically evaluated Mutode effectiveness by running it on 12 of the top 20 npm modules that have automated test suites.
- James H Andrews, Lionel C Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using mutation analysis for assessing and comparing testing coverage criteria. TSE 32, 8 (2006), 608–624. Google Scholar
Digital Library
- Babel. 2018. Babylon. https://www.npmjs.com/package/babylonGoogle Scholar
- Austin Bingham. 2017. Cosmic Ray: mutation testing for Python. https://github. com/sixty-north/cosmic-rayGoogle Scholar
- Ben Coe. 2018. Istanbul. https://istanbul.js.org/.Google Scholar
- Henry Coles. 2017. PIT. http://pitest.org/.Google Scholar
- Erik DeBill. 2018. Modulecounts. http://www.modulecounts.com/.Google Scholar
- Alex Denisov and Stanislav Pankevich. {n. d.}. Mull it over: mutation testing based on LLVM. https://github.com/mull-project/mull. ({n. d.}).Google Scholar
- Anna Derezińska and Piotr Trzpil. 2015. Mutation Testing Process Combined with Test-Driven Development in. NET Environment. In Theory and Engineering of Complex Systems and Dependability. 131–140.Google Scholar
- A. M. Fard and A. Mesbah. 2017. JavaScript: The (Un)Covered Parts. In ICST’17. 230–240.Google Scholar
- Node Foundation. 2018. ECMAScript Modules. https://nodejs.org/api/esm.html.Google Scholar
- Konrad Halas. 2017. MutPy: mutation testing tool for Python 3.x source code. https://github.com/mutpy/mutpyGoogle Scholar
- Quinn Hanam, Fernando S. de M. Brito, and Ali Mesbah. 2016. Discovering Bug Patterns in JavaScript. In ESEC/FSE’16. 144–156. Google ScholarDigital Library
- Ben Hartley. 2016. Mutant: A mutation testing framework for JavaScript. https: //github.com/benhartley/mutantGoogle Scholar
- TJ Holowaychuk. 2018. Mocha. https://mochajs.org/.Google Scholar
- Reyhaneh Jabbarvand and Sam Malek. 2017. muDroid: An Energy-aware Mutation Testing Framework for Android. In ESEC/FSE’17. 208–219. Google ScholarDigital Library
- Nico Jansen and Simon de Lang. 2018. Stryker: The JavaScript mutation testing framework. https://github.com/stryker-mutator/strykerGoogle Scholar
- Yue Jia and Mark Harman. 2008. MILU: A customizable, runtime-optimized higher order mutation testing tool for the full C language. In Practice and Research Techniques, 2008. TAIC PART’08. Testing: Academic & Industrial Conference. 94–98. Google ScholarDigital Library
- Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. TSE 37, 5 (2011), 649–678. Google ScholarDigital Library
- René Just. 2014. The Major mutation framework: Efficient and scalable mutation analysis for Java. In ISSTA’14. 433–436. Google ScholarDigital Library
- René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In ESEC/FSE 14. 654–665. Google ScholarDigital Library
- Vojta Jína. 2018. Karma. https://karma-runner.github.io/2.0/index.html.Google Scholar
- Mario Linares-Vásquez, Gabriele Bavota, Michele Tufano, Kevin Moran, Massimiliano Di Penta, Christopher Vendome, Carlos Bernal-Cárdenas, and Denys Poshyvanyk. 2017. Enabling mutation testing for Android apps. In ESEC/FSE’17. 233–244.Google Scholar
- Niels Lohmann. 2017. Mutate++: C++ Mutation Test Environment. https: //github.com/nlohmann/mutate_cppGoogle Scholar
- Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon. 2005. MuJava: An automated class mutation system. Software Testing, Verification and Reliability 15, 2 (2005), 97–133. Google ScholarCross Ref
- Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2013. Efficient JavaScript mutation testing. In ICST’13. 74–83. Google ScholarDigital Library
- Inc. npm. 2018. About npm. https://www.npmjs.com/aboutGoogle Scholar
- Inc. npm. 2018. Most depended upon packages. https://www.npmjs.com/browse/ dependedGoogle Scholar
- npm, Inc., Node.JS Foundation, and JS Foundation. 2018. Attitudes to security in the JavaScript community – npm, Inc. – Medium. https://medium.com/npm-inc/ security-in-the-js-community-4bac032e553bGoogle Scholar
- F. Ocariza, K. Bajaj, K. Pattabiraman, and A. Mesbah. 2013. An Empirical Study of Client-Side JavaScript Bugs. In ESEM’13. 55–64.Google Scholar
- Frolin S Ocariza Jr, Karthik Pattabiraman, and Benjamin Zorn. 2011. JavaScript errors in the wild: An empirical study. In ISSRE’11. 100–109. Google ScholarDigital Library
- A Jefferson Offutt, Jie Pan, Kanupriya Tewary, and Tong Zhang. 1996. An experimental evaluation of data flow and mutation testing. Softw., Pract. Exper. 26, 2 (1996), 165–176. Google ScholarDigital Library
- Maks Rafalko. 2018. Infection: PHP Mutation Testing Framework. https: //github.com/infection/infectionGoogle Scholar
- RequireJS. 2018. CommonJS. http://requirejs.org/docs/commonjs.html.Google Scholar
- Diego Rodríguez-Baquero. 2018. mutode. https://www.npmjs.com/package/ mutodeGoogle Scholar
- Diego Rodríguez-Baquero. 2018. Mutode - Mutators Documentation. https: //thesoftwaredesignlab.github.io/mutode/module-Mutators.htmlGoogle Scholar
- Diego Rodríguez-Baquero. 2018. Mutode: Mutation testing for JavaScript and Node.js. https://github.com/TheSoftwareDesignLab/mutodeGoogle Scholar
- Tony Roussel. 2016. NinjaTurtlesMutation. https://github.com/criteo/ NinjaTurtlesMutationGoogle Scholar
- Markus Schirp. 2018. Mutant: Mutation testing for Ruby. https://github.com/ mbj/mutantGoogle Scholar
- Michael G Schwern and Andy Lester. 2018. TAP specification. https://testanything. org/tap-specification.html.Google Scholar
- Daniel Tschinder, Logan Smyth, and Henry Zhu. 2018. Babel. https://babeljs.io/.Google Scholar
- Laurie Voss. 2018. Rolling weekly downloads of npm packages. https://twitter. com/seldo/status/988477780441481217 Abstract 1 Introduction 2 Related work 3 Mutode 3.1 Implemented Operations 3.2 Mutode Execution 3.3 Tool Usage and Extensibility 4 Evaluation 5 Demo remarks & Future Work ReferencesGoogle Scholar
Index Terms
- Mutode: generic JavaScript and Node.js mutation testing tool
Recommendations
Mitigating the effects of flaky tests on mutation testing
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisMutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail ...
Comments