Adaptive random testing with randomly translated failure region

ABSTRACT

Adaptive Random Testing (ART) algorithms are designed to be more effective than Random Testing. Some of these methods however distribute the test cases not evenly within the input domain. Therefore, some locations are preferred. Since the locations of failure-causing inputs of a system under test are obviously unkown, such a preference makes the method more effective for some systems under test and less effective for others. This paper addresses the described problem and tries to equalize the effectiveness of a testing method for all systems under test whose failure-causing inputs have the same geometric shape. Virtually, all failure-causing inputs are randomly translated to reach this goal. This method is applied to two well-known ART methods that tend to generate test cases at the corners and the boundary of the input domain more frequently. However, the presented method is not restricted to any strategy for test case selection.

REFERENCES

• 1: G. Casella and R. L. Berger. Statistical Inference. Wadsworth Group, Duxbury, CA, USA, 2002.
• 2: F. T. Chan, T. Y. Chen, I. K. Mak, and Y. T. Yu. Proportional sampling strategy: Guidelines for software testing practitioners. Information and Software Technology, 38:775--782, 1996.
• 3: Kwok Ping Chan , Tsong Yueh Chen , Fei-Ching Kuo , Dave Towey, A Revisit of Adaptive Random Testing by Restriction, Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04), p.78-85, September 28-30, 2004
• 4: Kwok Ping Chan , Tsong Yueh Chen , Dave Towey, Restricted Random Testing, Proceedings of the 7th International Conference on Software Quality, p.321-330, June 09-13, 2002
• 5: T. Y. Chen , R. Merkel , G. Eddy , P. K. Wong, Adaptive Random Testing Through Dynamic Partitioning, Proceedings of the Quality Software, Fourth International Conference on (QSIC'04), p.79-86, September 08-10, 2004 [doi>10.1109/QSIC.2004.6]
• 6: T. Y. Chen , D. H. Huang, Adaptive Random Testing by Localization, Proceedings of the 11th Asia-Pacific Software Engineering Conference (APSEC'04), p.292-298, November 30-December 03, 2004 [doi>10.1109/APSEC.2004.17]
• 7: T. Y. Chen and D. H. Huang. An improvement of adaptive random testing. Preprint, Swinburne University of Technology, Australia, 2006.
• 8: T. Y. Chen, D. H. Huang, and Z. Q. Zhou. Adaptive random testing through iterative partitioning. In Proceedings of the 11th Ada-Europe International Conference on Reliable Software Technologies (Ada-Europe 2006), volume 4006 of Lecture Notes in Computer Science, pages 155--166. Springer-Verlag, 2006.
• 9: Tsong Yueh Chen , Fei-Ching Kuo , Robert Merkel, On the Statistical Properties of the F-measure, Proceedings of the Quality Software, Fourth International Conference on (QSIC'04), p.146-153, September 08-10, 2004 [doi>10.1109/QSIC.2004.27]
• 10: T. Y. Chen, F.-C. Kuo, and R. Merkel. On the statistical properties of testing effectiveness measures. The Journal of Systems and Software, 79(5):591--601, 2006.
• 11: T. Y. Chen, F.-C. Kuo, R. G. Merkel, and S. P. Ng. Mirror adaptive random testing. Information and Software Technology, 46:1001--1010, 2004.
• 12: T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In M. J. Maher, editor, Proceedings of the 9th Asian Computing Science Conference (ASIAN 2004), volume 3321 of Lecture Notes in Computer Science, pages 320--329. Springer, 2004.
• 13: Tsong Yueh Chen , Robert Merkel, Quasi-random testing, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101957]
• 14: T. Y. Chen , Robert Merkel, Efficient and Effective Random Testing Using the Voronoi Diagram, Proceedings of the Australian Software Engineering Conference (ASWEC'06), p.300-299, April 18-21, 2006 [doi>10.1109/ASWEC.2006.25]
• 15: T. Y. Chen , T. H. Tse , Y. T. Yu, Proportional sampling strategy: a compendium and some insights, Journal of Systems and Software, v.58 n.1, p.65-81, August 2001 [doi>10.1016/S0164-1212(01)00028-0 ]
• 16: J. E. Forrester and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th USENIX Windows Systems Symposium, pages 59--68, 2000.
• 17: R. Hamlet. Random testing. In Encylopedia of Software Engineering, pages 970--978. Wiley, 1994.
• 18: I. K. Mak. On the effectiveness of random testing. Master thesis, Department of Computer Science, University of Melbourne Australia, 1997.
• 19: J. Mayer. Adaptive random testing by bisection with restriction. In Proceedings of the Seventh International Conference on Formal Engineering Methods (ICFEM 2005), volume 3785 of Lecture Notes in Computer Science, pages 251--263. Springer-Verlag, 2005.
• 20: Johannes Mayer, Lattice-based adaptive random testing, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101963]
• 21: J. Mayer. Adaptive random testing by bisection and localization. In Proceedings of the Fifth International Workshop on Formal Approaches to Testing of Software (FATES 2005), volume 3997 of Lecture Notes in Computer Science, pages 72--86. Springer-Verlag, 2006.
• 22: J. Mayer. Efficient and effective random testing based on partitioning and neighborhood. In Proceedings of the 18th International Conference on Software Engineering and Knowledge Engineering (SEKE 2006), 2006. (in press).
• 23: J. Mayer. Restricted adaptive random testing by random partitioning. In Proceedings of the International Conference on Software Engineering Research and Practice (SERP 2006). CSREA Press, 2006. (in press).
• 24: Johannes Mayer, Towards effective adaptive random testing for higher-dimensional input domains, Proceedings of the 8th annual conference on Genetic and evolutionary computation, July 08-12, 2006, Seattle, Washington, USA [doi>10.1145/1143997.1144323]
• 25: J. Mayer and C. Schneckenburger. Statistical analysis and enhancement of random testing methods also under constrained ressources. In Proceedings of the International Conference on Software Engineering Research and Practice (SERP 2006), 2006. (in press).
• 26: R. Merkel. Analysis and Enhancements of Adaptive Random Testing. PhD thesis, Swinburne University of Technology, Australia, 2005.
• 27: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 28: Donald R. Slutz, Massive Stochastic Testing of SQL, Proceedings of the 24rd International Conference on Very Large Data Bases, p.618-622, August 24-27, 1998
• 29: Takahide Yoshikawa , Kouya Shimura , Toshihiro Ozawa, Random Program Generator for Java JIT Compiler Test System, Proceedings of the Third International Conference on Quality Software, p.20, November 06-07, 2003

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS

General Terms:
Algorithms, Reliability, Verification

Keywords:
adaptive random testing, failure region, random testing, test case selection