Software testing with an operational profile: OP definition

Abstract

This article is devoted to the survey, analysis, and classification of operational profiles (OP) that characterize the type and frequency of software inputs and are used in software testing techniques. The survey follows a mixed method based on systematic maps and qualitative analysis. This article is articulated around a main dimension, that is, OP classes, which are a characterization of the OP model and the basis for generating test cases. The classes are organized as a taxonomy composed of common OP features (e.g., profiles, structure, and scenarios), software boundaries (which define the scope of the OP), OP dependencies (such as those of the code or in the field of interest), and OP development (which specifies when and how an OP is developed). To facilitate understanding of the relationships between OP classes and their elements, a meta-model was developed that can be used to support OP standardization. Many open research questions related to OP definition and development are identified based on the survey and classification.

References

1. S. Arora, R. B. Misra, and V. M. Kumre. 2005. Software reliability improvement through operational profile driven testing. In Proc. of the 53rd Annual Reliability and Maintainability Symposium. 621--627.
2. L. D. Bousquet, F. Ouabdesselam, and J.-L. Richier. 1998. Expressing and implementing operational profiles for reactive software validation. In Proc. of the 9th International Symposium on Software Reliability Engineering (ISSRE’98). 222--230.
3. P. A. Brooks and A. M. Memon. 2007. Automated GUI testing guided by usage profiles. In Proc. of the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE’07). 333--342.
4. K. Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage, Thousand Oaks, CA.
5. J. Corbin and A. Strauss. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory 3rd Ed. Sage, Thousand Oaks, CA.
6. D. Cramer. 1997. Basic Statistics for Social Research: Step-by-Step Calculations and Computer Techniques Using Minitab. Psychology Press.
7. C. Csallner and Y. Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Software Practice and Experience 34, 11, 1025--1050.
8. W. Dulz and F. Zhen. 2003. MaTeLo—Statistical usage testing by annotated sequence diagrams, Markov chains and TTCN-3. In Proc. of the 3rd International Conference on Quality Software. 336--342.
9. S. Elbaum and S. Narla. 2001. A methodology for operational profile refinement. In Proc. of the 2001 Annual Reliability and Maintainability Symposium (RAMS’01). 142--149.
10. D. F. Fabio, A. Carlomusto, A. Petrillo, and A. Ramondo. 2012. Human reliability analysis: A review of the state of the art. International Journal of Research in Management and Technology, 2, 1.
11. M. Gittens, H. Lutfiyya, and M. Bauer. 2004. An extended operational profile model. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 314--325.
12. B. G. Glaser and A. L. Strauss. 2009. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine de Gruyter, New York.
13. K. Gwet. 2002. Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical Methods for Inter-rater Reliability Assessment 1--6.
14. H. L. Guen and T. Thelin. 2003. Practical experiences with statistical usage testing. In Proc. of the 11th Annual Workshop on Software Test and Reliability Estimation Process at Software Technology and Engineering Practice (STEP’03). 87--93.
15. H. L. Guen, R. Marie, and T. Thelin. 2004. Reliability estimation for statistical usage testing using Markov chains. In Proc. of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 54--65.
16. D. Hamlet, D. Mason, and D. Woit. 2001. Theory of software reliability based on components. In Proc. of the 23rd International Conference on Software Engineering (ICSE’01). 361--370.
17. J. M. Hammersley and D. C. Handscomb. 1964. Monte Carlo Methods (Methuen's Monographs on Applied Probability and Statistics). Methuen, London.
18. D. Harel. 1987. Statecharts: A visual formalism for complex systems. Science of Computer Programming 8, 231--274.
19. B. Huang, M. Rodríguez, M. Li, J. Bernstein, and C. Smidts. 2011. Hardware error likelihood induced by the operation of software. IEEE Transactions on Reliability 60, 3, 622--639.
20. B. Huang, M. Rodríguez, M. Li, and C. Smidts. 2007. On the development of fault injection profiles. In Proc. of the 53nd Annual Reliability and Maintainability Symposium (RAMS’07). 226--231.
21. IEEE/ISO/IEC 24765. 2010. System and Software Engineering Vocabulary. Standards.
22. M. Jørgensen and M. Shepperd. 2007. A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering 33, 33--53.
23. B. D. Juhlin. 1992. Implementing operational profiles to measure system reliability. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’02). 286--295.
24. B. Kitchenham. 2004. Procedures for Performing Systematic Reviews. Technical Report. Keele University TR/SE-0401, NICTA Technical Report 0400011T.1, Keele University.
25. P. Koopman and J. Devale. 1999. Comparing the robustness of POSIX operating systems. In Proc. of the 29th IEEE International Symposium on Fault-Tolerant Computing. 30--37.
26. K. S. Kumar, R. B. Misra, and N. K. Goyal. 2008. Development of fuzzy software operational profile. International Journal of Reliability, Quality, and Safety Engineering 15, 581--597.
27. T. Muhr and S. Friese. 2004. User's Manual for ATLAS. ti 6.0. ATLAS. ti Scientific Software Development GmbH, Berline.
28. J. D. Musa. 1992. The operational profile in software reliability engineering: An overview. In Proc. of the 3rd International Symposium on Software Reliability Engineering (ISSRE’92). 140--154.
29. J. D. Musa. 1993. Operational profiles in software-reliability engineering. IEEE Software 10, 2, 14--32.
30. F. Ouabdesselam and I. Parissis. 1995. Constructing operational profiles for synchronous critical software. In Proc. of the 6th International Symposium on Software Reliability Engineering (ISSRE’95). 286--293.
31. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. 71--80.
32. S. J. Prowell. 2003. JUMBL: A tool for model-based statistical testing. In Proc. of the 36th Hawaii International Conference on System Sciences (HICSS’03).
33. M. Popovic and J. Kovacevic. 2007. A statistical approach to model-based robustness testing. In Proc. of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07). 485--494.
34. A. Rosenberg and E. Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proc. of HLT-NAACL 2004: Short Papers. Association for Computational Linguistics, 77--80.
35. P. Runeson and C. Wohlin. 1993. Statistical usage testing for software reliability certification and control. In Proc. of the 1st European International Conference on Software Testing, Analysis and Review (EuroSTAR’93). 309--323
36. J. Saldana. 2012. The Coding Manual for Qualitative Researchers (No. 14). Sage.
37. B. Shneideman and C. Plaisant. 2010. Designing the User Interface: Strategies for Effective Human Interaction. Addison-Wesley.
38. R. Shukla, D. Carrington, and P. Strooper. 2004a. Systematic operational profile development for software components. In Proc. of the 11th Asia-Pacific Software Engineering Conference (APSEC’04). 528--537.
39. R. Y. Shukla, P. A. Strooper, and D. A. Carrington. 2004b. A framework for reliability assessment of software components. In Proc. of the 7th International Symposium on Component-based Software Engineering (CBSE7). 272--279.
40. J. Sim and C. C. Wright. 2005. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy 85, 3, 257--268.
41. C. Taylor, T. C. Gibbs and A. Lewins. 2005. Quality of Qualitative Analysis. Retrieved from http://onlineqda.hud.ac.uk/Intro_QDA/qualitative_analysis.php.
42. G. H. Walton, J. H. Poore, and C. J. Trammell. 1995. Statistical testing of software based on a usage model. Software—Practice and Experience 25, 1, 97--108.
43. Y. Wei, M. Rodríguez, and C. Smidts. 2010. PRA framework for software propagation analysis of failures. Journal of Risk and Reliability 224, 2, 113--135.
44. E. J. Weyuker. 1998. Testing component-based software: A cautionary tale. IEEE Software 15, 5, 54--59.
45. J. A. Whittaker and J. Voas. 2000. Toward a more reliable theory of software reliability. IEEE Computer 33, 12, 36--42.
46. J. A. Whittaker and J. H. Poore. 1993. Markov analysis of software specifications. ACM Transactions on Software Engineering and Methodology. 2, 1, 93--106.
47. J. A. Whittaker. and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Transactions on Software Engineering 20, 10, 812--824.
48. D. M. Woit. 1993. Specifying Operational Profiles for Modules. In Proc. of the 1993 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’93). 2--10.
49. J. Yan, J. Wang, and H. W. Chen. 2004. Automatic generation of Markov chain usage models from real-time software UML Models. In Proc. of the 4th International Conference on Quality Software (QSIC’04). 22--31.
50. F. Zhen and C. Peng. 2004. A system test methodology based on the Markov chain usage model. In Proc. of the 8th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2004). 160--165.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Reliability

    • Validation

• Mathematics of computing

  • Probability and statistics

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Empirical software validation

      • Formal software verification

      • Process validation

      • Software defect analysis

        • Software testing and debugging

  • Software organization and properties

    • Extra-functional properties

      • Software reliability

    • Software functional properties

      • Formal methods

• Theory of computation

  • Semantics and reasoning

    • Program reasoning

      • Program verification