Analyzing the validity of selective mutation with dominator mutants

ABSTRACT

Various forms of selective mutation testing have long been accepted as valid approximations to full mutation testing. This paper presents counterevidence to traditional selective mutation. The recent development of dominator mutants and minimal mutation analysis lets us analyze selective mutation without the noise introduced by the redundancy inherent in traditional mutation. We then exhaustively evaluate all small sets of mutation operators for the Proteum mutation system and determine dominator mutation scores and required work for each of these sets on an empirical test bed. The results show that all possible selective mutation approaches have poor dominator mutation scores on at least some of these programs. This suggests that to achieve high performance with respect to full mutation analysis, selective approaches will have to become more sophisticated, possibly by choosing mutants based on the specifics of the artifact under test, that is, specialized selective mutation.

References

1. A. T. Acree. On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta, Georgia, USA, 1980.
2. Paul Ammann, Marcio E. Delamaro, and Jeff Offutt. Establishing theoretical minimal sets of mutants. In 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), pages 21–31, Cleveland, Ohio, USA, March 2014.
3. E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for C. Software Testing, Verification, and Reliability, Wiley, 11(2):113–136, June 2001.
4. T. A. Budd. Mutation Analysis of Program Test Data. PhD thesis, Yale University, New Haven, Connecticut, USA, 1980.
5. T. Chusho. Test data selection and quality estimation based on the concept of essential branches for path testing. IEEE Transactions on Software Engineering, 13(5), May 1987.
6. M. E. Delamaro, J. C. Maldonado, M. Jino, and M. Chaim. Proteum: Uma ferramenta de teste baseada na análise de mutantes (Proteum: A testing tool based on mutation analysis). In 7th Brazilian Symposium on Software Engineering, pages 31–33, Rio de Janeiro, Brazil, October 1993. In Portuguese.
7. Marcio E. Delamaro, Lin Deng, Serapilha Dureli, Nan Li, and Jeff Offutt. Experimental evaluation of SDL and one-op mutation for C. In 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 2014.
8. Márcio E. Delamaro, Lin Deng, Nan Li, and Vinicius H. S. Durelli. Growing a reduced set of mutation operators. In Proceedings of the 2014 Brazilian Symposium on Software Engineering (SBES), pages 81–90, Maceió, Alagoas, Brazil, September-October 2014.
9. Richard A. DeMillo, Richard J. Lipton, and Fred G. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4):34–41, April 1978.
10. Richard A. DeMillo and Jeff Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):900–910, September 1991.
11. Lin Deng, Jeff Offutt, and Nan Li. Empirical evaluation of the statement deletion mutation operator. In 6th IEEE International Conference on Software Testing, Verification and Validation (ICST 2013), pages 65–74, Luxembourg, March 2013.
12. R. Gopinath, Alipour A., Ahmed I., C. Jensen, and A. Groce. How hard does mutation analysis have to be anyway? In IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), Nov 2015.
13. Rahul Gopinath, Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. Do mutation reduction strategies matter? Technical report, School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA, August 2015.
14. Bernard Grün, David Schuler, and Andreas Zeller. The impact of equivalent mutants. In Fifth IEEE Workshop on Mutation Analysis (Mutation 2009), pages 192–199, Denver, Colorado, USA, April 2009.
15. M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE ’94), pages 191–200, Sorrento, Italy, May 1994. IEEE Computer Society Press.
16. René Just, Gregory M. Kapfhammer, and Franz Schweiggert. Do redundant mutants affect the effectiveness and efficiency of mutation analysis? In Eighth IEEE Workshop on Mutation Analysis (Mutation 2012), Montreal, Canada, April 2012.
17. René Just and Franz Schweiggert. Higher accuracy and lower run time: Efficient mutation analysis using non-redundant mutation operators. Journal of Software Testing, Verification and Reliability, 25(5-7):490–507, August-November 2015.
18. Garrett Kaminski, Paul Ammann, and Jeff Offutt. Better predicate testing. In Sixth Workshop on Automation of Software Test (AST 2011), pages 57–63, Honolulu HI, USA, May 2011.
19. Garrett Kaminski, Paul Ammann, and Jeff Offutt. Improving logic-based testing. Journal of Systems and Software, Elsevier, 86:2002–2012, August 2013.
20. Bob Kurtz, Paul Ammann, Marcio E. Delamaro, Jeff Offutt, and Lin Deng. Mutation subsumption graphs. In Tenth IEEE Workshop on Mutation Analysis (Mutation 2014), pages 176–185, Cleveland, Ohio, USA, March 2014.
21. Bob Kurtz, Paul Ammann, and Jeff Offutt. Static analysis of mutant subsumption. In Eleventh IEEE Workshop on Mutation Analysis (Mutation 2015), Graz, Austria, April 2015.
22. Bob Kurtz, Paul Ammann, Jeff Offutt, and Mariet Kurtz. Are we there yet? How redundant and equivalent mutants affect determination of test completeness. In Twelfth IEEE Workshop on Mutation Analysis (Mutation 2016), Chicago, Illinois, USA, April 2016.
23. Birgitta Lindström and András Márki. On strong mutation and subsuming mutants. In Twelfth IEEE Workshop on Mutation Analysis (Mutation 2016), Chicago, Illinois, USA, April 2016.
24. Yu-Seung Ma, Jeff Offutt, and Yong-Rae Kwon. MuJava: A mutation system for Java. In Proceedings of the 28th International Conference on Software Engineering (ICSE ’06), tool demo, pages 827–830, Shanghai, China, May 2006. IEEE Computer Society Press.
25. Aditya Mathur. Performance, effectiveness, and reliability issues in software testing. In Proceedings of the Fifteenth Annual International Computer Software and Applications Conference, pages 604–605, Tokyo, Japan, September 1991.
26. Akbar Siami Namin and James H. Andrews. Finding sufficient mutation operators via variable reduction. In Second IEEE Workshop on Mutation Analysis (Mutation 2006), Raleigh, NC, November 2006.
27. Akbar Siami Namin and James H. Andrews. On sufficiency of mutants. In Proceedings of the 29th International Conference on Software Engineering, Doctoral Symposium, pages 73–74, Minneapolis, MN, USA, 2007. ACM.
28. Akbar Siami Namin, James H. Andrews, and Duncan J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In Proceedings of the 30th International Conference on Software Engineering, pages 351–360, Chicago, Illinois, USA, 2008. ACM.
29. Jeff Offutt, Ammei Lee, Gregg Rothermel, Roland Untch, and Christian Zapf. An experimental determination of sufficient mutation operators. ACM Transactions on Software Engineering Methodology, 5(2):99–118, April 1996.
30. Jeff Offutt, Gregg Rothermel, and Christian Zapf. An experimental evaluation of selective mutation. In Proceedings of the Fifteenth International Conference on Software Engineering, pages 100–107, Baltimore, MD, May 1993. IEEE Computer Society Press.
31. Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon. Threats to the validity of mutation-based test assessment. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’16), Saarbrücken, Germany, July 2016, to appear.
32. David Schuler and Andreas Zeller. (un-)covering equivalent mutants. In 3rd IEEE International Conference on Software Testing, Verification and Validation (ICST 2010), pages 45–54, Paris, France, April 2010.
33. O. Schutze, X. Esquivel, A. Lara, and C. A. C. Coello. Using the averaged Hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE Transactions on Evolutionary Computation, 16(4):504–522, August 2012.
34. Roland Untch. On reduced neighborhood mutation analysis using a single mutagenic operator. In ACM Southeast Regional Conference, pages 19–21, Clemson SC, March 2009.
35. W. Eric Wong, M. E. Delamaro, J. C. Maldonado, and Aditya P. Mathur. Constrained mutation in C programs. In Proceedings of the 8th Brazilian Symposium on Software Engineering, pages 439–452, Curitiba, Brazil, October 1994.
36. W. Eric Wong and Aditya P. Mathur. Reducing the cost of mutation testing: An empirical study. Journal of Systems and Software, Elsevier, 31(3):185–196, December 1995.
37. W. Eric Wong and Aditya P. Mathur. Reducing the cost of mutation testing: An empirical study. Journal of Systems and Software, 31(3):185–196, December 1995.
38. Xiangjuan Yao, Mark Harman, and Yue Jia. A study of equivalent and stubborn mutation operators using human analysis of equivalence. In Proceedings of the 36th International Conference on Software Engineering, pages 919–930, Hyderabad, India, May 2014.
39. Lu Zhang, Shan-San Hou, Jun-Jue Hu, Tao Xie, and Hong Mei. Is operator-based mutant selection superior to random mutant selection? In 32nd ACM/IEEE International Conference on Software Engineering, pages 435–444, May 2010.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging