Does automated white-box test generation really help software testers?

ABSTRACT

Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

References

1. S. Afshan, P. McMinn, and M. Stevenson. Evolving readable string test inputs using a natural language model to reduce human oracle cost. In Int. Conference on Software Testing, Verification and Validation (ICST), 2013. (To appear).
2. A. Arcuri and L. Briand. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability. (To appear).
3. L. Baresi, P. L. Lanzi, and M. Miraz. Testful: an evolutionary test approach for Java. In IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 185–194, 2010.
4. C. Csallner and Y. Smaragdakis. JCrasher: an automatic robustness tester for Java. Software: Practice and Experience, 34(11):1025–1050, 2004.
5. J. T. de Souza, C. L. Maia, F. G. de Freitas, and D. P. Coutinho. The human competitiveness of search based software engineering. In International Symposium on Search Based Software Engineering (SSBSE), pages 143–152, 2010.
6. G. Fraser and A. Arcuri. Evosuite: Automatic test suite generation for object-oriented software. In ACM Symposium on the Foundations of Software Engineering (FSE), pages 416–419, 2011.
7. G. Fraser and A. Arcuri. Sound empirical evidence in software testing. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 178–188, 2012.
8. G. Fraser and A. Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering, 39(2):276–291, 2013.
9. G. Fraser and A. Zeller. Exploiting common object usage in test case generation. In IEEE International Conference on Software Testing, Verification and Validation (ICST), pages 80–89. IEEE Computer Society, 2011.
10. G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering, 28(2):278–292, 2012.
11. M. Harman, S. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR), 45(1):11, 2012.
12. M. Harman and P. McMinn. A theoretical and empirical study of search based testing: Local, global and hybrid search. IEEE Transactions on Software Engineering, 36(2):226–247, 2010.
13. M. Islam and C. Csallner. Dsc+mock: A test case + mock class generator in support of coding against interfaces. In International Workshop on Dynamic Analysis (WODA), pages 26–31, 2010.
14. R. Just, F. Schweiggert, and G. Kapfhammer. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In International Conference on Automated Software Engineering (ASE), pages 612–615, 2011.
15. B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El Emam, and J. Rosenberg. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721–734, 2002.
16. J. Koza. Human-competitive results produced by genetic programming. Genetic Programming and Evolvable Machines, 11(3):251–284, 2010.
17. K. Lakhotia, P. McMinn, and M. Harman. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. Journal of Systems and Software, 83(12):2379–2391, 2010.
18. P. McMinn. Search-based software test data generation: A survey. Software Testing, Verification and Reliability, 14(2):105–156, 2004.
19. E. F. Miller, Jr. and R. A. Melton. Automated generation of testcase datasets. In International Conference on Reliable Software, pages 51–58. ACM, 1975.
20. A. Namin and J.H.Andrews. The influence of size and coverage on test suite effectiveness. In ACM International Symposium on Software Testing and Analysis (ISSTA). ACM, 2009.
21. C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for Java. In Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), pages 815–816. ACM, 2007.
22. C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 199–209, 2011.
23. C. Pasareanu and N. Rungta. Symbolic pathfinder: symbolic execution of java bytecode. In IEEE/ACM International Conference on Automated Software Engineering (ASE), volume 10, pages 179–180, 2010.
24. F. Pastore, L. Mariani, and G. Fraser. Crowdoracles: Can the crowd solve the oracle problem? In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2013. (To appear).
25. R. Ramler, D. Winkler, and M. Schmidt. Random test case generation and manual unit testing: Substitute or complement in retrofitting tests for legacy code? In EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), pages 286–293. IEEE, 2012.
26. G. Sautter, K. Böhm, F. Padberg, and W. Tichy. Empirical evaluation of semi-automated XML annotation of text documents with the GoldenGATE editor. Research and Advanced Techn. for Digital Libraries, pages 357–367, 2007.
27. C. Seaman. Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering, 25(4):557–572, 1999.
28. D. Sjoberg, J. Hannay, O. Hansen, V. By Kampenes, A. Karahasanovic, N. Liborg, and A. C Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733–753, 2005.
29. M. Staats, G. Gay, and M. Heimdahl. Automated oracle creation support, or: how I learned to stop worrying about fault propagation and love mutation testing. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 870–880, 2012.
30. M. Staats, S. Hong, M. Kim, and G. Rothermel. Understanding user understanding: determining correctness of generated program invariants. In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 188–198. ACM, 2012.
31. N. Tillmann and N. J. de Halleux. Pex — white box test generation for .NET. In International Conference on Tests And Proofs (TAP), pages 134–253, 2008.
32. P. Tonella. Evolutionary testing of classes. In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 119–128, 2004.
33. J. Wegener, A. Baresel, and H. Sthamer. Evolutionary test environment for automatic structural testing. Information and Software Technology, 43(14):841–854, 2001.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging