Zalia Shams
Stephen H. Edwards
ABSTRACT
Many educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage metrics assess what percentage of code has been exercised, but do not depend on whether a test suite adequately checks that the expected behavior is achieved. This paper evaluates checked coverage, an alternative measure of test thoroughness aimed at overcoming this limitation, along with object branch coverage, a structure code coverage metric that has received little discussion in educational assessment. Checked coverage works backwards from behavioral assertions in test cases, measuring the dynamic slice of the executed code that actually influences the outcome of each assertion. Object branch coverage (OBC) is a stronger coverage criterion similar to weak variants of modified condition/decision coverage. We experimentally compare checked coverage and OBC against statement coverage, branch coverage, mutation analysis, and all-pairs testing to evaluate which is the best predictor of how likely a test suite is to detect naturally occurring defects. While checked coverage outperformed other coverage measures in our experiment, followed closely by OBC, both were only weakly correlated with a test suite's ability to detect naturally occurring defects produced by students in the final versions of their programs. Still, OBC appears to be an improved and practical alternative to existing statement and branch coverage measures, while achieving nearly the same benefits as checked coverage.
- K. Aaltonen, P. Ihantola, and O. Seppälä, "Mutation analysis vs. code coverage in automated assessment of students' testing skills," presented at the Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion, Reno/Tahoe, Nevada, USA, 2010. Google Scholar
Digital Library
- M. Bordin, C. Comar, T. Gingold, J. Guitton, O. Hainque, and T. Quinot. Object and Source Coverage for Critical Applications with the COUVERTURE Open Analysis Framework. In ERTS (Embedded Real Time Sofware and Systems Conference), May 2010.Google Scholar
- J. J. Chilenski, "An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion" U.S. Department of Transportation DOT/FAA/AR-01/18, 2001.Google Scholar
- C. Comar, J. Guitton, O. Hainque, T. Quinot. Formalization and Comparison of MCDC and Object Branch Coverage Criteria. In ERTS (Embedded Real Time Software and Systems Conference), May 2012.Google Scholar
- (9/5/2014). Clover: Java and Groovy Code Coverage. Available: https://www.atlassian.com/software/clover/overviewGoogle Scholar
- S. H. Edwards and Z. Shams, "Comparing Test Quality Measures for Assessing Student-Written Tests," in 36th International Conference on Software Engineering, Hyderabad, India, 2014, p. to apprear. Google ScholarDigital Library
- S. H. Edwards, Z. Shams, M. Cogswell, and R. C. Senkbeil, "Running students' software tests against each others' code: new life for an old "gimmick"," presented at the Proceedings of the 43rd ACM technical symposium on Computer Science Education, Raleigh, North Carolina, USA, 2012. Google ScholarDigital Library
- (9/5/2014). JaCoCo Java Code Coverage Library. Available: http://www.eclemma.org/jacoco/Google Scholar
- (9/5/2014). JavaSlicer. Available: http://www.st.cs.uni-saarland.de/javaslicer/Google Scholar
- D. Schuler and A. Zeller, "Assessing Oracle Quality with Checked Coverage," presented at the Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, 2011. Google ScholarDigital Library
- Z. Shams and S. H. Edwards, "Toward practical mutation analysis for evaluating the quality of student-written software tests," presented at the Proceedings of the ninth annual international ACM conference on International computing education research, San Diego, San California, USA, 2013. Google ScholarDigital Library
- M. D. Weiser, "Program slices: formal, psychological, and practical investigations of an automatic program abstraction method," University of Michigan, 1979.Google Scholar
- H. Zhu, P. A. V. Hall, and J. H. R. May, "Software Unit Test Coverage and Adequacy", ACM Computing Surveys, 29(4): 366--427, December 1997. Google ScholarDigital Library
Index Terms
- Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
Recommendations
Toward practical mutation analysis for evaluating the quality of student-written software tests
ICER '13: Proceedings of the ninth annual international ACM conference on International computing education researchSoftware testing is being added to programming courses at many schools, but current assessment techniques for evaluating student-written tests are imperfect. Code coverage measures are typically used in practice, but they have limitations and sometimes ...
Comparing test quality measures for assessing student-written tests
ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software EngineeringMany educators now include software testing activities in programming assignments, so there is a growing demand for appropriate methods of assessing the quality of student-written software tests. While tests can be hand-graded, some educators also use ...
Do student programmers all tend to write the same software tests?
ITiCSE '14: Proceedings of the 2014 conference on Innovation & technology in computer science educationWhile many educators have added software testing practices to their programming assignments, assessing the effectiveness of student-written tests using statement coverage or branch coverage has limitations. While researchers have begun investigating ...
Comments