MD-ART: a test case generation method without test oracle problem

ABSTRACT

Adaptive random testing (ART), as an improved random testing method, preserves the advantages of traditional random test method and overcomes the blindness of traditional random testing method. But it is usually not easy to validate the correctness of the output, except for some special test cases. In other words, the test oracle problem is unresolved. In this research, we introduced metamorphic testing (MT) and metamorphic distance into ART, which is called metamorphic distance based ART (MD-ART) to provide the test oracle. The results of primary experiment results show that MD-ART performs better than traditional MT and ART not only in test effectiveness but also in test efficiency and test coverage.

References

1. E.T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo, The Oracle Problem in Software Testing: A Survey, IEEE Transactions on Software Engineering, 41 (2015) 507-525.
2. E.J. Weyuker, The Oracle Assumption of Program Testing, in: Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, 1980, pp. 44-49
3. T.Y. Chen, S.C. Cheung, S. Yiu, Metamorphic testing: a new approach for generating next test cases, Department of Computer Science, Hong Kong University of Science and Technology, Tech. Rep. HKUST-CS98-01, (1998).
4. J. Zhang, J.-j. Chen, D. Hao, Y.-f. Xiong, B. Xie, L. Zhang, H. Mei, Search-Based Inference of Polynomial Metamorphic Relations, in: Proceedings of the 29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014), 2014, pp. 701-712.
5. U. Kanewala, Techniques for Automatic Detection of Metamorphic Relations, in: Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, 2014, pp. 237-238.
6. G. Singh, An Automated Metamorphic Testing Technique for Designing Effective Metamorphic Relations, in: Contemporary Computing, Springer, 2012, pp. 152-163.
7. Y. Cao, Z.Q. Zhou, T.Y. Chen, On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions, in: Quality Software (QSIC), 2013 13th International Conference on, IEEE, 2013, pp. 153-162.
8. L. Huai, K. Fei-Ching, D. Towey, C. Tsong Yueh, How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?, Software Engineering, IEEE Transactions on, 40 (2014) 4-22.
9. J. Mayer, R. Guderlei, An empirical study on the selection of good metamorphic relations, in: Computer Software and Applications Conference, 2006. COMPSAC'06. 30th Annual International, IEEE, 2006, pp. 475-484.
10. Z.W. Hui, S. Huang, A Formal Model for Metamorphic Relation Decomposition, in: 2013 Fourth World Congress on Software Engineering, Hong Kong, China, 2013, pp. 64- 68.
11. W.E. Wong, V. Debroy, A survey of software fault localization, Department of Computer Science, University of Texas at Dallas, Tech. Rep. UTDCS-45-09, (2009).
12. T.Y. Chen, F.-C. Kuo, Y. Liu, A. Tang, Metamorphic Testing and Testing with Special Values, in: SNPD, 2004, pp. 128- 134.
13. P. Wu, X.-C. Shi, J.-J. Tang, H.-M. Lin, T.Y. Chen, Metamorphic Testing and Special Case Testing: A Case Study, Journal of Software, 16 (2005) 1210-1220.
14. M. Jiang, T.Y. Chen, F.-C. Kuo, Z. Ding, Testing Central Processing Unit scheduling algorithms using Metamorphic Testing, in: Software Engineering and Service Science (ICSESS), 2013 4th IEEE International Conference on, IEEE, 2013, pp. 530-536.
15. A. Singh, S. Kang, Metamorphic Testing: Using the Properties of Sut, International Journal of Computer Technology and Applications 02 (2011) 1334-1336.
16. S. Yoo, Metamorphic testing of stochastic optimisation, in: Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, IEEE, 2010, pp. 192-201.
17. F.-C. Kuo, S. Liu, T.Y. Chen, Testing a binary space partitioning algorithm with metamorphic testing, in: Proceedings of the 2011 ACM Symposium on Applied Computing, ACM, 2011, pp. 1482-1489.
18. T.Y. Chen, F.-C. Kuo, R.G. Merkel, T.H. Tse, Adaptive Random Testing: The ART of test case diversity, J. Syst. Softw., 83 (2010) 60-66.
19. T.Y. Chen, H. Leung, I. Mak, Adaptive random testing, in: Advances in Computer Science-ASIAN 2004. Higher-Level Decision Making, Springer, 2005, pp. 320-329.
20. K.P. Chan, T.Y. Chen, F.-C. Kuo, D. Towey, A revisit of adaptive random testing by restriction, in: Computer Software and Applications Conference, 2004. COMPSAC 2004. Proceedings of the 28th Annual International, IEEE, 2004, pp. 78-85.
21. T.Y. Chen, R. Merkel, P. Wong, G. Eddy, Adaptive random testing through dynamic partitioning, in: Quality Software, 2004. QSIC 2004. Proceedings. Fourth International Conference on, IEEE, 2004, pp. 79-86.
22. T.Y. Chen, F.-C. Kuo, R.G. Merkel, T. Tse, Adaptive random testing: The art of test case diversity, Journal of Systems and Software, 83 (2010) 60-66.
23. T.Y. Chen, R. Merkel, An upper bound on software testing effectiveness, ACM Transactions on Software Engineering and Methodology (TOSEM), 17 (2008) 16.
24. R. Merkel, Analysis and enhancements of adaptive random testing, Ph. D. Thesis, (2005).
25. P. Wu, Iterative metamorphic testing, in: Computer Software and Applications Conference, 2005. COMPSAC 2005. 29th Annual International, IEEE, 2005, pp. 19-24.
26. G.W. Dong, C.H. Nie, B.W. Xu, L.L. Wang, An effective iterative metamorphic testing algorithm based on program path analysis, in: Quality Software, 2007. QSIC'07. Seventh International Conference on, IEEE, 2007, pp. 292-297.
27. L. Chen, L. Cai, J. Liu, Z. Liu, S. Wei, P. Liu, An optimized method for generating cases of metamorphic testing, in: Information Science and Service Science and Data Mining (ISSDM), 2012 6th International Conference on New Trends in, Taipei, Taiwan, China, 2012, pp. 439 - 443.
28. F.T. Chan, T.Y. Chen, I.K. Mak, Y.T. Yu, Proportional sampling strategy: guidelines for software testing practitioners, Information and Software Technology, 38 (1996) 775-782.
29. Z.Y. Zhang, W.K. Chan, T.H. Tse, P.F. Hu, Experimental study to compare the use of metamorphic testing and assertion checking, Journal of Software, 20 (2009) 2637- 2654.
30. T.Y. Chen, F.-C. Kuo, R. Merkel, On the statistical properties of testing effectiveness measures, Journal of Systems and Software, 79 (2006) 591-601.
31. R.A. DeMillo, R.J. Lipton, F.G. Sayward, Hints on test data selection: Help for the practicing programmer, Computer, 11 (1978) 34-41.
32. H. Zhu, P.A. Hall, J.H. May, Software unit test coverage and adequacy, ACM Computing Surveys (CSUR), 29 (1997) 366-427.
33. W.H. Press, Numerical recipes 3rd edition: The art of scientific computing, Cambridge university press, 2007.
34. G. Dong, Metamorphic testing techniques for error detection efficiency, in: School of Computer Science and Engineering, Southeast University, Nanjing, China, 2009.
35. M. Hutchins, H. Foster, T. Goradia, T. Ostrand, Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria, in: Proceedings of the 16th international conference on Software engineering, IEEE Computer Society Press, 1994, pp. 191-200.
36. G. License, Gcov: Gnu coverage tool, in.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging