Improving software testing by observing practice

ABSTRACT

The objective of this qualitative study was to understand the complex practice of software testing, and based on this knowledge, to develop process improvement propositions that could concurrently reduce development and testing costs and improve software quality. First, a survey of testing practices was onducted and 26 organizational units (OUs) were interviewed. From this sample, five OUs were further selected for an in-depth case study. The study used grounded theory as its research method and the data was collected from 41 theme-based interviews. The analysis yielded improvement propositions that included enhanced testability of software components, efficient communication and interaction between development and testing, early involvement of testing, and risk-based testing. The connective and central improvement proposition was that testing ought to adapt to the business orientation of the OU. Other propositions were integrated around this central proposition. The results of this study can be used in improving development and testing processes.

REFERENCES

• 1: ATLAS.ti - The Knowledge Workbench: Scientific Software Development, 2005.
• 2: Pekka Abrahamsson, Commitment development in software process improvement: critical misconceptions, Proceedings of the 23rd International Conference on Software Engineering, p.71-80, May 12-19, 2001, Toronto, Ontario, Canada
• 3: Ståle Amland, Risk-based testing: risk analysis fundamentals and metrics for software testing including a financial application case study, Journal of Systems and Software, v.53 n.3, p.287-295, Sep 15, 2000 [doi>10.1016/S0164-1212(00)00019-4 ]
• 4: Richard Baskerville , Linda Levine , Jan Pries-Heje , Balasubramaniam Ramesh , Sandra Slaughter, How Internet Software Companies Negotiate Quality, Computer, v.34 n.5, p.51-57, May 2001 [doi>10.1109/2.920612 ]
• 5: Kai-Yuan Cai , T. Y. Chen , Yong-Chao Li , Wei-Yi Ning , Y. T. Yu, Adaptive testing of software components, Proceedings of the 2005 ACM symposium on Applied computing, March 13-17, 2005, Santa Fe, New Mexico [doi>10.1145/1066677.1067011]
• 6: Carter, C. R. and Dresner, M. Purchasing's role in environmental management: Cross-functional development of grounded theory. Journal of Supply Chain Management, 37, 3 (2001), 12--28.
• 7: Cynthia F. Cohen , Stanley J. Birkin , Monica J. Garfield , Harold W. Webb, Managing conflict in software testing, Communications of the ACM, v.47 n.1, p.76-81, January 2004 [doi>10.1145/962081.962083]
• 8: Eisenhardt, K. M. Building Theories from Case Study Research. Academy of Management Review, 14, 4 (1989), 532--550.
• 9: EU SME Definition: European Commission, 2003.
• 10: Glaser, B. and Strauss, A. L. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, Chicago, 1967.
• 11: Dorothy Graham, REQUIREMENTS: Requirements and Testing: Seven Missing-Link Myths, IEEE Software, v.19 n.5, p.15-17, September 2002 [doi>10.1109/MS.2002.1032845 ]
• 12: Mary Jean Harrold, Testing: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.61-72, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336532]
• 13: Donald E. Harter , Sandra A. Slaughter, Process maturity and software quality: a field study, Proceedings of the twenty first international conference on Information systems, p.407-411, December 2000, Brisbane, Queensland, Australia
• 14: ISO/IEC ISO/IEC 15504-1, Information Technology - Process Assessment - Part 1: Concepts and Vocabulary. 2002.
• 15: Barbara A. Kitchenham , Shari Lawrence Pfleeger , Lesley M. Pickard , Peter W. Jones , David C. Hoaglin , Khaled El Emam , Jarrett Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, v.28 n.8, p.721-734, August 2002 [doi>10.1109/TSE.2002.1027796 ]
• 16: Heinz K. Klein , Michael D. Myers, A set of principles for conducting and evaluating interpretive field studies in information systems, MIS Quarterly, v.23 n.1, p.67-93, March 1999 [doi>10.2307/249410]
• 17: Ravindra Krovi , Akhilesh Chandra , Balaji Rajagopalan, Information flow parameters for managing organizational processes, Communications of the ACM, v.46 n.2, p.77-82, February 2003 [doi>10.1145/606272.606277]
• 18: Miles, M. B. and Huberman, A. M. Qualitative Data Analysis. SAGE Publications, Thousand Oaks, CA, 1994.
• 19: S. P. Ng , T. Murnane , K. Reed , D. Grant , T. Y. Chen, A Preliminary Survey on Software Testing Practices in Australia, Proceedings of the 2004 Australian Software Engineering Conference (ASWEC'04), p.116, April 13-16, 2004
• 20: Northrop, L. Software Product Lines: Reuse That Makes Business sense, Software Engineering Institute, Carnegie Mellon University, 2006.
• 21: Leon J. Osterweil, Software processes are software too, revisited: an invited talk on the most influential paper of ICSE 9, Proceedings of the 19th international conference on Software engineering, p.540-548, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253440]
• 22: G. Paré , J. J. Elam, Using case study research to build theories of IT implementation, Proceedings of the IFIP TC8 WG 8.2 international conference on Information systems and qualitative research, p.542-568, January 1997, Philadelphia, Pennsylvania, United States
• 23: Salminen, V., Yassine, A., and Riitahuhta, A. Strategic Management of Complexity in Distributed Product Development. In NordDesign 2000, (Copenhagen, 2000).
• 24: Carolyn B. Seaman, Qualitative Methods in Empirical Studies of Software Engineering, IEEE Transactions on Software Engineering, v.25 n.4, p.557-572, July 1999 [doi>10.1109/32.799955 ]
• 25: Sandra A. Slaughter , Donald E. Harter , Mayuram S. Krishnan, Evaluating the cost of software quality, Communications of the ACM, v.41 n.8, p.67-73, Aug. 1998 [doi>10.1145/280324.280335]
• 26: Smolander, K., Rossi, M., and Purao, S. Going beyond the Blueprint: Unraveling the Complex Reality of Software Architectures. In the 13th European Conference on Information Systems: Information Systems in a Rapidly Changing Economy, (Ragensburg, Germany, 2005).
• 27: Strauss, A. and Corbin, J. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. SAGE Publications, Newbury Park, CA, 1990.
• 28: Taipale, O., Smolander, K., and Kälviäinen, H. Cost Reduction and Quality Improvement in Software Testing. In Software Quality Management Conference, (Southampton, UK, 2006).
• 29: Ossi Taipale , Heikki Kalviainen , Kari Smolander, Factors Affecting Software Testing Time Schedule, Proceedings of the Australian Software Engineering Conference (ASWEC'06), p.283-291, April 18-21, 2006 [doi>10.1109/ASWEC.2006.27]
• 30: Taipale, O., Smolander, K., and Kälviäinen, H. A Survey on Software Testing. In 6th International SPICE Conference on Software Process Improvement and Capability dEtermination (SPICE'2006), (Luxembourg, 2006).
• 31: Richard Torkar , Stefan Mankefors, A Survey on Testing and Reuse, Proceedings of the IEEE International Conference on Software-Science, Technology & Engineering, p.164, November 04-05, 2003

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Productivity

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
      D.2.5 Testing and Debugging

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management

Keywords:
grounded theory, process improvement, software testing