skip to main content
10.1145/1159733.1159763acmconferencesArticle/Chapter ViewAbstractPublication PagesesemConference Proceedingsconference-collections
Article

A literature survey of the quality economics of defect-detection techniques

Published: 21 September 2006 Publication History

Abstract

Over the last decades, a considerable amount of empirical knowledge about the efficiency of defect-detection techniques has been accumulated. Also a few surveys have summarised those studies with different focuses, usually for a specific type of technique. This work reviews the results of empirical studies and associates them with a model of software quality economics. This allows a better comparison of the different techniques and supports the application of the model in practice as several parameters can be approximated with typical average values. The main contributions are the provision of average values of several interesting quantities w.r.t. defect detection and the identification of areas that need further research because of the limited knowledge available.

References

[1]
E. N. Adams. Optimizing Preventive Service of Software Products. IBM Journal of Research and Development, 28(1):2--14, 1984.
[2]
V. R. Basili and R.W. Selby. Comparing the Effectiveness of Software Testing Strategies. IEEE Transactions on Software Engineering, SE-13(12):1278--1296, 1987.
[3]
B. Boehm. Software Engineering Economics. Prentice Hall, 1981.
[4]
L. Briand, K. E. Emam, O. Laitenberger, and T. Fussbroich. Using Simulation to Build Inspection Efficiency Benchmarks for Development Projects. In Proc. 20th International Conference on Software Engineering (ICSE'98), pages 340--349. IEEE Computer Society, 1998.
[5]
J. Christmansson and P. Santhanam. Error Injection Aimed at Fault Removal in Fault Tolerance Mechanisms -- Criteria for Error Selection using Field Data on Software Faults. In Proc. Seventh International Symposium on Software Reliability Engineering (ISSRE'96), pages 175--184. IEEE Computer Society, 1996.
[6]
M. E. Fagan. Reviews and Inspections. In M. Broy and E. Denert, editors, Software Pioneers -- Contributions to Software Engineering, pages 562--573. Springer, 2002.
[7]
C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended Static Checking for Java. In Proc. 2002 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 234--245. ACM Press, 2002.
[8]
T. Gilb and D. Graham. Software Inspection. Addison-Wesley, 1993.
[9]
R. B. Grady and T. Van Slack. Key Lessons in Achieving Widespread Inspection Use. IEEE Software, 11(4):46--57, 1994.
[10]
W. E. Howden. Theoretical and Empirical Studies of Program Testing. IEEE Transactions on Software Engineering, SE-4(4):293--298, 1978.
[11]
R. Johnson and D. Wagner. Finding User/Kernel Pointer Bugs With Type Inference. In Proc. 13th USENIX Security Symposium, pages 119--134, 2004.
[12]
C. Jones. Applied Software Measurement: Assuring Productivity and Quality. McGraw-Hill, 1991.
[13]
N. Juristo, A. M. Moreno, and S. Vegas. Reviewing 25 Years of Testing Technique Experiments. Empirical Software Engineering, 9:7--44, 2004.
[14]
N. Juristo and S. Vegas. Functional Testing, Structural Testing and Code Reading: What Fault Type Do They Each Detect. In R. Conradi and A. I. Wang, editors, Empirical Methods and Studies in Software Engineering: Experiences from ESERNET, volume 2765 of LNCS, pages 208--232. Springer, 2003.
[15]
O. Laitenberger. A Survey of Software Inspection Technologies. In Handbook on Software Engineering and Knowledge Engineering, volume 2, pages 517--555. World Scientific Publishing, 2002.
[16]
B. Littlewood, P. T. Popov, L. Strigini, and N. Shryane. Modeling the Effects of Combining Diverse Software Fault Detection Techniques. IEEE Transactions on Software Engineering, 26(12):1157--1167, 2000.
[17]
R. R. Lutz and I. C. Mikulski. Empirical Analysis of Safety-Critical Anomalies During Operations. IEEE Transactions on Software Engineering, 30(3):172--180, 2004.
[18]
G. J. Myers. The Art of Software Testing. John Wiley & Sons, 1979.
[19]
P. M. Nagel, F. W. Scholz, and J. A. Skrivan. Software Reliability: Additional Investigations into Modeling with Replicated Experiments. NASA Contractor Rep. 172378, NASA Langley Res. Center, Jun. 1984.
[20]
S. C. Ntafos. On Comparisons of Random, Partition, and Proportional Partition Testing. IEEE Transactions on Software Engineering, 27(10):949--960, 2001.
[21]
A. Rai, H. Song, and M. Troutt. Software Quality Assurance: An Analytical Survey and Research Prioritization. Journal of Systems and Software, 40:67--83, 1998.
[22]
R. J. Rubey. Quantitative Aspects of Software Validation. In Proc. International Conference on Reliable Software, pages 246--251. ACM Press, 1975.
[23]
M. Sullivan and R. Chillarege. A Comparison of Software Defects in Database Management Systems and Operating Systems. In Proc. 22nd International Symposium on Fault-Tolerant Computing (FTCS-22), pages 475--484. IEEE Computer Society, 1992.
[24]
S. Wagner. A Literature Survey of the Software Quality Economics of Defect-Detection Techniques. Technical Report TUM-I0614, Institut für Informatik, Technische Universität München, 2006.
[25]
S. Wagner. A Model and Sensitivity Analysis of the Quality Economics of Defect-Detection Techniques. In Proc. International Symposium on Software Testing and Analysis (ISSTA'06). ACM Press, 2006.
[26]
S. Wagner. Cost-Optimisation of Analytical Software Quality Assurance. PhD Dissertation, Technische Universität München, 2006. To appear.
[27]
S. Wagner and H. Fischer. A Software Reliability Model Based on a Geometric Sequence of Failure Rates. In Proc. 11th International Conference on Reliable Software Technologies (Ada-Europe'06), volume 4006 of LNCS. Springer, 2006.
[28]
S. Wagner, J. Jürjens, C. Koller, and P. Trischberger. Comparing Bug Finding Tools with Reviews and Tests. In Proc. 17th International Conference on Testing of Communicating Systems (TestCom'05), volume 3502 of LNCS, pages 40--55. Springer, 2005.

Cited By

View all
  • (2024)Metrics for software process simulation modelingJournal of Software: Evolution and Process10.1002/smr.2676Online publication date: 11-Jul-2024
  • (2023)(Nothing But) Many Eyes Make All Bugs ShallowProceedings of the 2023 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses10.1145/3605770.3625216(53-63)Online publication date: 30-Nov-2023
  • (2023)Automated NFR testing in continuous integration environments: a multi-case study of Nordic companiesEmpirical Software Engineering10.1007/s10664-023-10356-128:6Online publication date: 24-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISESE '06: Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
September 2006
388 pages
ISBN:1595932186
DOI:10.1145/1159733
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cost/benefit
  2. defectdetection techniques
  3. literature survey
  4. quality cost
  5. software quality economics

Qualifiers

  • Article

Conference

ISESE06
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)14
  • Downloads (Last 6 weeks)2
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Metrics for software process simulation modelingJournal of Software: Evolution and Process10.1002/smr.2676Online publication date: 11-Jul-2024
  • (2023)(Nothing But) Many Eyes Make All Bugs ShallowProceedings of the 2023 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses10.1145/3605770.3625216(53-63)Online publication date: 30-Nov-2023
  • (2023)Automated NFR testing in continuous integration environments: a multi-case study of Nordic companiesEmpirical Software Engineering10.1007/s10664-023-10356-128:6Online publication date: 24-Oct-2023
  • (2022)Continuously Managing NFRs: Opportunities and Challenges in PracticeIEEE Transactions on Software Engineering10.1109/TSE.2021.306633048:7(2629-2642)Online publication date: 1-Jul-2022
  • (2021)A Paired Learner-Based Approach for Concept Drift Detection and Adaptation in Software Defect PredictionApplied Sciences10.3390/app1114666311:14(6663)Online publication date: 20-Jul-2021
  • (2021)Software Project Management Using Machine Learning Technique—A ReviewApplied Sciences10.3390/app1111518311:11(5183)Online publication date: 2-Jun-2021
  • (2021)Automated Visual Testing of Application User Interfaces Using Static Analysis of ScreenshotsInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402150004231:02(167-191)Online publication date: 2-Mar-2021
  • (2021)Software fault prediction based on the dynamic selection of learning technique: findings from the eclipse project studyApplied Intelligence10.1007/s10489-021-02346-x51:12(8945-8960)Online publication date: 16-Apr-2021
  • (2021)Verifikation und Integration von SoftwareEinführung in die Softwaretechnik10.1007/978-3-662-50263-1_12(463-534)Online publication date: 1-Jul-2021
  • (2020)The Lack of Shared Understanding of Non-Functional Requirements in Continuous Software Engineering: Accidental or Essential?2020 IEEE 28th International Requirements Engineering Conference (RE)10.1109/RE48521.2020.00021(90-101)Online publication date: Aug-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media