Adaptive random prioritization for interaction test suites

ABSTRACT

Combinatorial interaction testing (CIT), a black-box testing method, has been well studied in recent years. It aims at constructing an effective interaction test suites, so as to identify the faults that are caused by interactions among parameters. After interaction test suites are generated by CIT, the execution order of test cases in the test suite becomes critical due to limited testing resources. To determine test case order, the prioritization of interaction test suites has been employed. As we know, random prioritization (RP) of test cases has been considered as simple but ineffective. Existing research unveils that adaptive random prioritization (ARP) of test cases is an alternative and promising candidate that may replace RP. However, previous ARP techniques may not be used to prioritize interaction test suites due to the lack of source-code-related information in interaction test suite, such as statement coverage, function coverage, or branch coverage. In this paper, we not only propose the ARP strategy in order to prioritize interaction test suites by using interaction coverage information, without the source-code-related information, but also unify the RP strategy and traditional interaction-coverage based prioritization strategy (ICBP). Additionally, simulation studies indicate that the ARP strategy performs better than the RP strategy, test-case-generation prioritization, and reverse test-case-generation prioritization, and can also be more time-saving than ICBP while greatly maintaining similar, or even better, effectiveness.

References

1. R. C. Bryce and C. J. Colbourn. Test prioritization for pairwise interaction coverage. In Proceedings of the 1st International Workshop on Advances in Model-based Testing, pages 1--7, St. Louis, Missouri, USA, 2005.
2. R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pairwise coverage with seeding and contraints. Information and Software Technology, 48(10): 960--970, October 2006.
3. R. C. Bryce and A. M. Memon. Test suite prioritization by interaction coverage. In Proceedings of the Workshop on Domain Specific Approaches to Software Test Automation, pages 1--7, Dubrovnik, Croatia, 2007.
4. T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference, pages 320--329, Chiang Mai, Thailand, 2004.
5. J. Czerwonka. Pairwise testing in real world: Practical extensions to test case generators. In Proceedings of the 24th Pacific Northwest Software Quality Conference, pages 419--430, Portland, USA, 2006.
6. R. Huang, J. Chen, T. Zhang, R. Wang, and Y. Lu. Prioritizing variable-strength covering array. In Proceedings of the 37th IEEE Annual Computer Software and Applications Conference, pages 502--601, Kyoto Japan, 2013.
7. R. Huang, X. Xie, T. Y. Chen, and Y. Lu. Adaptive random test case generation for combinatorial testing. In Proceedings of the 36th IEEE Annual Computer Software and Applications Conference, pages 52--61, Izmir, Turkey, 2012.
8. R. Huang, X. Xie, D. Towey, T. Y. Chen, Y. Lu, and J. Chen. Prioritization of combinatorial test cases by incremental interaction coverage. International Journal of Software Engineering and Knowledge Engineering, To appear, 2013.
9. B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse. Adaptive random test case prioritization. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering, pages 233--244, Auckland, New Zealand, 2009.
10. Y. Lei, R. Kacker, D. R. Kuhn, and V. Okun. Ipog/ipod: Efficient test generation for multi-way software testing. Software Testing, Verification, and Reliability, 18(3): 125--148, September 2008.
11. Z. Li, M. Harman, and R. Hierons. Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering, 33(4): 225--237, April 2007.
12. C. Nie and H. Leung. A survey of combinatorial testing. ACM Computer Survey, 43(2): 11:1--11:29, January 2011.
13. X. Qu, M. B. Cohen, and K. M. Woolf. Combinatorial interaction regression testing: A study of test case generation and prioritization. In Proceedings of the 23rd International Conference on Software Maintenance, pages 255--264, Paris, France, 2007.
14. G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10): 929--948, October 2001.
15. Z. Wang, L. Chen, B. Xu, and Y. Huang. Cost-cognizant combinatorial test case prioritization. International Journal of Software Engineering and Knowledge Engineering, 21(6): 829--854, September 2011.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging