Testing and inspecting reusable product line components: first empirical results

ABSTRACT

In recent years, product line development has increasingly received attention in industry as it enables software-developing organizations to reduce both cost and time of developing and maintaining increasingly complex systems as well as to address the demands for individually customized products. Successful product line development requires high quality of reusable artifacts in order to achieve the promised benefits. The unique issues of quality assurance in the context of systematic reuse, however, have not been quantitatively investigated so far. This paper describes a first empirical study comparing the two defect detection techniques, code inspections and functional testing, in the context of product line development. The primary goal of the study was to initially investigate the defect finding potential of the techniques on reusable software components with common and variant features. The major findings of the study are that the two techniques identified different types of defects on variants of a reusable component. Inspections are on average 66.39% more effective and need on average 36.84% less effort to detect a defect We found that both the testing and inspection techniques applied in the experiment were ineffective in identifying variant-specific defects. Overall, the results indicate that the standard quality assurance techniques seem to be insufficient to address special characteristics of reusable components.

REFERENCES

• 1: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 2: Ram Chillarege , Inderpal S. Bhandari , Jarir K. Chaar , Michael J. Halliday , Diane S. Moebus , Bonnie K. Ray , Man-Yuen Wong, Orthogonal Defect Classification-A Concept for In-Process Measurements, IEEE Transactions on Software Engineering, v.18 n.11, p.943-956, November 1992 [doi>10.1109/32.177364 ]
• 3: Software product lines: practices and patterns, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2001
• 4: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 5: Victor R. Basili, Evolving and packaging reading technologies, Journal of Systems and Software, v.38 n.1, p.3-12, July 1997 [doi>10.1016/S0164-1212(97)00065-4 ]
• 6: Fagan, M. E. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal, 15(3):182--211, 1976.
• 7: Geppert, B., Krueger, C., and Li, J. (eds.). Proceedings of the International Workshop on Software Product Line Testing (SPLiT 2004), Boston, MA, Aug. 2004.
• 8: Geppert, B., Krueger, C., and Trew, T. (eds.). Proceedings of the Second International Workshop on Software Product Line Testing (SPLiT 2005), Rennes, France, Sep. 2005.
• 9: Tom Gilb , Dorothy Graham , Susannah Finzi, Software Inspection, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1993
• 10: William C. Hetzel, An experimental analysis of program verification methods., 1976
• 11: IFIP WG 10.4, "Dependability: Basic Concepts and Terminology," IFIP Working Group on Dependable Computing and Fault Tolerance, October 1990.
• 12: Juristo, N., Vegas, S., Functional testing, structural testing and code reading: what fault type do they each detect?, Empirical Methods and Studies in Software Engineering -- Experiences from ESERNET, 2003.
• 13: Erik Kamsties , Christopher M. Lott, An Empirical Evaluation of Three Defect-Detection Techniques, Proceedings of the 5th European Software Engineering Conference, p.362-383, September 25-28, 1995
• 14: Kolb, R., and Muthig, D. Challenges in Testing Software Product Lines. In Proceedings of CONQUEST'03, Nuremberg, Germany, Sep. 2003, pp. 81--95.
• 15: Kolb, R., McGregor, J. D., and Muthig, D. Proceedings of the First International Workshop on Quality Assurance in Reuse Contexts (QUARC), Boston, MA, Fraunhofer IESE Report No. 096.04/E, Aug. 2004.
• 16: Laitenberger, O. Cost effective Detection of Software Defects through Perspective-based Inspections, PhD Theses in Experimental Software Engineering, 2000.
• 17: McGregor, J. D. Testing a Software Product Line. Technical Report, CMU/SEI-2001-TR-022, Software Engineering Institute, Carnegie Mellon University, Dec. 2001.
• 18: Glenford J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Communications of the ACM, v.21 n.9, p.760-768, Sept. 1978 [doi>10.1145/359588.359602]
• 19: Peer reviews in software: a practical guide, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 20: Wood, W., Roper, M., Brooks, A. and Miller, J. Comparing and Combining Software Defect Detection Techniques: A Replicated Empirical Study. In 6th European Software Engineering Conference, pp. 262--277. 1997.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Code inspections and walk-throughs

General Terms:
Experimentation, Measurement

Keywords:
controlled experiment, functional testing, inspection, quality assurance, reusable components, software product line