Test purpose generation in an industrial application

ABSTRACT

Nowadays test engineers use various strategies for the design of test cases. Among others, test cases are designed on basis of structural coverage criteria or test cases are related to specific fault models. In this paper we evaluate these two techniques for test purpose design. We present a heuristic algorithm for the extraction of test cases from TGV's output, i.e., the test process. We discuss the problem of overlapping test purposes and illustrate improvements in terms of test execution time and in terms of number of test cases when minimizing this overlap. Furthermore, we present different strategies for the generation of fault-based test purposes. For our evaluation we apply the presented techniques to a Session Initiation Protocol (SIP) Registrar specification. All extracted test cases are executed against a commercial and an open source implementation of such a SIP Registrar.

REFERENCES

• 1: B. K. Aichernig and C. C. Delgado. From faults via test purposes to test cases: On the fault-based testing of concurrent systems. In FASE, volume 3922 of LNCS, pages 324--338. Springer, 2006.
• 2: Bernhard K. Aichernig , Bernhard Peischl , Martin Weiglhofer , Franz Wotawa, Protocol Conformance Testing a SIP Registrar: an Industrial Application of Formal Methods, Proceedings of the Fifth IEEE International Conference on Software Engineering and Formal Methods, p.215-226, September 10-14, 2007 [doi>10.1109/SEFM.2007.31]
• 3: Lydie du Bousquet , Solofo Ramangalahy , Séverine Simon , César Viho , Axel Belinfante , René G. de Vries, Formal Test Automation: The Conference Protocol with TGV/TORX, Proceedings of the IFIP TC6/WG6.1 13th International Conference on Testing Communicating Systems: Tools and Techniques, p.221-228, August 29-September 01, 2000
• 4: Jean-Claude Fernandez , Claude Jard , Thierry Jéron , César Viho, An experiment in automatic generation of test suites for protocols with verification technology, Science of Computer Programming, v.29 n.1-2, p.123-146, July 1997 [doi>10.1016/S0167-6423(96)00032-9 ]
• 5: J. Franks , P. Hallam-Baker , J. Hostetler , S. Lawrence , P. Leach , A. Luotonen , L. Stewart, HTTP Authentication: Basic and Digest Access Authentication, RFC Editor, 1999
• 6: H. Garavel, F. Lang, and R. Mateescu. An overview of CADP 2001. European Association for Software Science and Technology Newsletter, 4:13--24, 2002.
• 7: J. Grabowski, D. Hogrefe, and R. Nahm. Test case generation with test purpose specification by MSC's. In SDL'93, the 6th SDL Forum, pages 253--266. Elsevier Science, 1993.
• 8: Wolfgang Grieskamp , Nikolai Tillmann , Colin Campbell , Wolfram Schulte , Margus Veanes, Action Machines - Towards a Framework for Model Composition, Exploration and Conformance Testing Based on Symbolic Computation, Proceedings of the Fifth International Conference on Quality Software, p.72-82, September 19-20, 2005 [doi>10.1109/QSIC.2005.10]
• 9: ISO. ISO 8807: Information processing systems - open systems interconnection - LOTOS - a formal description technique based on the temporal ordering of observational behaviour, 1989.
• 10: Claude Jard , Thierry Jéron, TGV: theory, principles and algorithms: A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems, International Journal on Software Tools for Technology Transfer (STTT), v.7 n.4, p.297-315, August 2005 [doi>10.1007/s10009-004-0153-x]
• 11: Hakim Kahlouche , César Viho , Massimo Zendri, An Industrial Experiment in Automatic Generation of Executable Test Suites for a Cache Coherency Protocol, Proceedings of the IFIP TC6 11th International Workshop on Testing Communicating Systems, p.211-226, August 31-September 02, 1998
• 12: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 13: J. Rosenberg , H. Schulzrinne , G. Camarillo , A. Johnston , J. Peterson , R. Sparks , M. Handley , E. Schooler, SIP: Session Initiation Protocol, RFC Editor, 2002
• 14: J. Tretmans. Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools, 17(3):103--120, 1996.
• 15: J. Tretmans and E. Brinksma. TorX: Automated model based testing. In A. Hartman and K. Dussa-Zieger, editors, Proceedings of the First European Conference on Model-Driven Software Engineering, pages 13--25, Nurnburg, Germany, 2003.
• 16: M. Weiglhofer. A LOTOS formalization of SIP. Technical Report SNA-TR-2006-1P1, Competence Network Softnet Austria, December 2006.
• 17: M. Weiglhofer. Conformance testing of a session initiation protocol server. Technical Report SNA-TR-2006-1P2, Competence Network Softnet Austria, To appear.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification

General Terms:
Experimentation, Reliability, Verification

Keywords:
fault-based test purposes, model-based testing, protocol conformance testing, test purpose design