Practical selective regression testing with effective redundancy in interleaved tests

ABSTRACT

As software systems evolve and change over time, test suites used for checking the correctness of software typically grow larger. Together with size, test suites tend to grow in redundancy. This is especially problematic for complex highly-configurable software domains, as growing the size of test suites significantly impacts the cost of regression testing.

In this paper we present a practical approach for reducing ineffective redundancy of regression suites in continuous integration testing (strict constraints on time-efficiency) for highly-configurable software. The main idea of our approach consists in combining coverage based redundancy metrics (test overlap) with historical fault-detection effectiveness of integration tests, to identify ineffective redundancy that is eliminated from a regression test suite. We first apply and evaluate the approach in testing of industrial video conferencing software. We further evaluate the approach using a large set of artificial subjects, in terms of fault-detection effectiveness and timeliness of regression test feedback. We compare the results with an advanced retest-all approach and random test selection. The results show that regression test selection based on coverage and history analysis can: 1) reduce regression test feedback compared to industry practice (up to 39%), 2) reduce test feedback compared to the advanced retest-all approach (up to 45%) without significantly compromising fault-detection effectiveness (less than 0.5% on average), and 3) improve fault detection effectiveness compared to random selection (72% on average).

References

1. A. Andrews, S. Elakeili, and A. Alhaddad. 2015. Selective Regression Testing of Safety-Critical Systems: A Black Box Approach. In Software Quality, Reliability and Security - Companion (QRS-C), 2015 IEEE International Conference on. 22--31.
2. H. Baller, S. Lity, M. Lochau, and I. Schaefer. 2014. Multi-objective Test Suite Optimization for Incremental Product Family Testing. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 303--312.
3. K. Z. Bell and M. A. Vouk. 2005. On effectiveness of pairwise methodology for testing network-centric software. In 2005 International Conference on Information and Communication Technology. 221--235.
4. Yanping Chen, Robert L. Probert, and D. Paul Sims. 2002. Specification-based Regression Test Selection with Risk Analysis. In Proc. of the 2002 Conf. of the Centre for Advanced Studies on Collaborative Research. IBM Press. http://dl.acm.org/citation.cfm?id=782115.782116
5. Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 235--245.
6. Emelie Engstrom and Per Runeson. 2013. Test Overlay in an Emerging Software Product Line - An Industrial Case Study. Information and Software Technology 55, 3 (March 2013), 581--594.
7. Gordon Fraser and Franz Wotawa. 2007. Redundancy Based Test-suite Reduction. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE'07). Springer-Verlag, Berlin, Heidelberg, 291--305. http://dl.acm.org/citation.cfm?id=1759394.1759425
8. M. Gligoric, L. Eloussi, and D. Marinov. 2015. Ekstazi: Lightweight Test Selection. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. 713--716.
9. Arnaud Gotlieb and Dusica Marijan. 2014. FLOWER: Optimal Test Suite Reduction As a Network Maximum Flow. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 171--180.
10. Rajiv Gupta, M. Jean Harrold, and Mary Lou Soffa. 1996. Program Slicing-Based Regression Testing Techniques. Software Testing, Verification and Reliability 6, 2 (1996), 83--111.
11. M. Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A Methodology for Controlling the Size of a Test Suite. ACM Transactions on Software Engineering and Methodology 2, 3 (July 1993), 270--285.
12. M. P. E. Heimdahl and D. George. 2004. Test-suite reduction for model based tests: effects on test quality and implications for testing. In International Conference on Automated Software Engineering 2004. 176--185.
13. Aymeric Hervieu, Dusica Marijan, Arnaud Gotlieb, and Benoit Baudry. 2016. Practical Minimization of Pairwise-covering Test Configurations Using Constraint Programming. Information Software Technology 71, C (March 2016), 129--146.
14. H. Y. Hsu and A. Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In International Conference on Software Engineering.
15. D. Jeffrey and Neelam Gupta. 2005. Test suite reduction with selective redundancy. In 21st IEEE International Conference on Software Maintenance (ICSM'05). 549--558.
16. Dennis Jeffrey and Neelam Gupta. 2007. Improving Fault Detection Capability by Selectively Retaining Test Cases During Test Suite Reduction. IEEE Transactions on Software Engineering 33, 2 (Feb. 2007), 108--123.
17. Jung-Min Kim and Adam Porter. 2002. A History-based Test Prioritization Technique for Regression Testing in Resource Constrained Environments. In International Conference on Software Engineering. ACM, New York, NY, USA, 119--129.
18. Negar Koochakzadeh and Vahid Garousi. 2010. A Tester-assisted Methodology for Test Redundancy Detection. Advanc. Software Engineering 2010, Article 6 (Jan. 2010), 13 pages.
19. N. Koochakzadeh, V. Garousi, and F. Maurer. 2009. Test Redundancy Measurement Based on Coverage Information: Evaluations and Lessons Learned. In International Conference on Software Testing Verification and Validation. 220--229.
20. B. Korel, L. H. Tahat, and B. Vaysburg. 2002. Model based regression test reduction using dependence analysis. In Software Maintenance, 2002. Proceedings. International Conference on. 214--223.
21. Chengying Mao and Yansheng Lu. 2005. Regression testing for component-based software systems by enhancing change information. In 12th Asia-Pacific Software Engineering Conference (APSEC'05). 8 pp.-.
22. D. Marijan, A. Gotlieb, and S. Sen. 2013. Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study. In International Conference on Software Maintenanse 2013. 540--543.
23. D. Marijan and M. Liaaen. 2016. Effect of Time Window on the Performance of Continuous Regression Testing. In International Conference on Software Maintenance and Evolution. 568--571.
24. W. Masri, A. Podgurski, and D. Leon. 2007. An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows. IEEE Transactions on Software Engineering 33, 7 (July 2007), 454--477.
25. A. Jefferson Offutt, Jie Pan, and Jeffrey M. Voas. 1995. Procedures for reducing the size of coverage-based test sets. In International Conference Testing Computer Software. 111--123.
26. Alessandro Orso, Taweesup Apiwattanapong, and Mary Jean Harrold. 2003. Leveraging Field Data for Impact Analysis and Regression Testing. In Proceedings of the 9th European Software Engineering Conference (ESEC/FSE-11). ACM, New York, NY, USA, 128--137.
27. Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. Chianti: A Tool for Change Impact Analysis of Java Programs. In Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA '04). ACM, New York, NY, USA, 432--448.
28. Gregg Rothermel and Mary Jean Harrold. 1997. A Safe, Efficient Regression Test Selection Technique. ACM Transactiona on Software Engineering and Methodology 6, 2 (April 1997), 173--210.
29. G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In International Conference on Software Maintenance. 34--43.
30. Gregg Rothermel, Mary Jean Harrold, Jeffery von Ronne, and Christie Hong. 2002. Empirical Studies of Test-Suite Reduction. Journal of Software Testing, Verification, and Reliability 12 (2002), 219--249.
31. A. S. M. Sajeev and B. Wibowo. 2003. Regression test selection based on version changes of components. In Software Engineering Conference, 2003. Tenth Asia-Pacific. 78--85.
32. Amanda Schwartz and Hyunsook Do. 2016. Cost-effective regression testing through Adaptive Test Prioritization strategies. Journal of Systems and Software 115 (2016), 61 -- 81.
33. Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz Khurshid. 2011. An Empirical Study of JUnit Test-Suite Reduction. In Proceedings of the 2011 IEEE 22Nd International Symposium on Software Reliability Engineering (ISSRE '11). IEEE Computer Society, Washington, DC, USA, 170--179.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Empirical software validation