Improving student performance by evaluating how well students test their own programs

ABSTRACT

Students need to learn more software testing skills. This paper presents an approach to teaching software testing in a way that will encourage students to practice testing skills in many classes and give them concrete feedback on their testing performance, without requiring a new course, any new faculty resources, or a significant number of lecture hours in each course where testing will be practiced. The strategy is to give students basic exposure to test-driven development, and then provide an automated tool that will assess student submissions on-demand and provide feedback for improvement. This approach has been demonstrated in an undergraduate programming languages course using a prototype tool. The results have been positive, with students expressing appreciation for the practical benefits of test-driven development on programming assignments. Experimental analysis of student programs shows a 28% reduction in defects per thousand lines of code.

REFERENCES

• 1: Eric Allen , Robert Cartwright , Charles Reis, Production programming in the classroom, Proceedings of the 34th SIGCSE technical symposium on Computer science education, February 19-23, 2003, Reno, Navada, USA
• 2: Sigal Ar , Jin-Yi Cai, Reliable benchmarks using numerical instability, Proceedings of the fifth annual ACM-SIAM symposium on Discrete algorithms, p.34-43, January 23-25, 1994, Arlington, Virginia, United States
• 3: Bagert, D., Hilburn, T., Hislop, G., Lutz, M., Mccracken, M., and Mengel, S. 1999. Guidelines for Software Engineering Education Version 1.0, Technical Report CMU/SEI-99-TR-032, Software Engineering Institute, Pittsburg, PA.
• 4: Kent Beck, Aim, Fire, IEEE Software, v.18 n.5, p.87-89, September 2001 [doi>10.1109/52.951502 ]
• 5: Beck, Test Driven Development: By Example, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 6: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 7: Boehm, B. 1976. Software engineering. IEEE Transactions on Computers, C-25, 12, 1226-1241.
• 8: Bourque, P., Dupuis, R., Abran, A., and Moore, J.W., eds. 2001. Guide to the Software Engineering Body of Knowledge--Stone Man Trial Version 1.00, IEEE Computer Society, Washington, available at: <http:/www.swebok.org>.
• 9: Duane Buck , David J. Stucki, Design early considered harmful: graduated exposure to complexity and structure based on levels of cognitive development, Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.75-79, March 07-12, 2000, Austin, Texas, United States
• 10: Checkstyle. 2003. Checkstyle home page. Web page last accessed Mar. 21, 2003. <http://checkstyle.sourceforge.net/>.
• 11: Cortex, Inc. 2003. Clover: a code coverage tool for Java. Web page accessed Mar. 21, 2003. <http://www.thecortex.net/clover/>.
• 12: Stephen H. Edwards, Rethinking computer science education from a test-first perspective, Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 26-30, 2003, Anaheim, CA, USA [doi>10.1145/949344.949390]
• 13: Michael H. Goldwasser, A gimmick to integrate software testing throughout the curriculum, Proceedings of the 33rd SIGCSE technical symposium on Computer science education, February 27-March 03, 2002, Cincinnati, Kentucky
• 14: Mary Jean Harrold, Testing: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.61-72, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336532]
• 15: Thomas B. Hilburn , Massood Townhidnejad, Software quality: a curriculum postscript?, Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.167-171, March 07-12, 2000, Austin, Texas, United States
• 16: Julia Isong, Developing an automated program checkers, Journal of Computing Sciences in Colleges, v.16 n.3, p.218-224, March 2001
• 17: David Jackson , Michelle Usher, Grading student programs using ASSYST, Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education, p.335-339, February 27-March 01, 1997, San Jose, California, United States
• 18: Edward L. Jones, Software testing in the computer science curriculum -- a holistic approach, Proceedings of the Australasian conference on Computing education, p.153-157, December 2000, Melbourne, Australia [doi>10.1145/359369.359392]
• 19: Jones, E.L 2000b. SPRAE: A framework for teaching software testing in the undergraduate curriculum. In Proc. ADMI 2000, Hampton, VA, 1-4 June 2000.
• 20: Edward L. Jones, Integrating testing into the curriculum — arsenic in small doses, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.337-341, February 2001, Charlotte, North Carolina, United States
• 21: Jones, E.L. 2001b. An experiential approach to incorporating software testing into the computer science curriculum. In Proc. 2001 Frontiers in Education Conf. (FiE 2001), F3D7-F3D11.
• 22: Edward L. Jones, Grading student programs - a software testing approach, Journal of Computing Sciences in Colleges, v.16 n.2, p.185-192, Jan. 2001
• 23: JUnit. 2003. JUnit home page. Web page last accessed Mar. 21, 2003. <http://www.junit.org/>.
• 24: Michael Luck , Mike Joy, A secure on-line submission system, Software—Practice & Experience, v.29 n.8, p.721-740, July 10, 1999 [doi>10.1002/(SICI)1097-024X(19990710)29:8<721::AID-SPE257>3.3.CO;2-S ]
• 25: Nachiappan Nagappan , Laurie Williams , Miriam Ferzli , Eric Wiebe , Kai Yang , Carol Miller , Suzanne Balik, Improving the CS1 experience with pair programming, Proceedings of the 34th SIGCSE technical symposium on Computer science education, February 19-23, 2003, Reno, Navada, USA
• 26: Renée A. McCauley , Clark Archer , Nell Dale , Rym Mili , James Robergé , Harriet Taylor, The effective integration of software engineering principles throughout the undergraduate computer science curriculum (abstract), Proceedings of the twenty-sixth SIGCSE technical symposium on Computer science education, p.364-365, March 02-04, 1995, Nashville, Tennessee, United States
• 27: Reneée McCauley , Nell Dale , Thomas Hilburn , Susan Mengel , Branson W. Murrill, The assimilation of software engineering into the undergraduate computer science curriculum (panel session), Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.423-424, March 07-12, 2000, Austin, Texas, United States
• 28: McQuain, W. 2003. Curator: An electronic submission management environment. Web page last accessed July 24, 2003. <http://www.cs.vt.edu/curator/>.
• 29: Susan A. Mengel , Vinay Yerramilli, A case study of the static analysis of the quality of novice student programs, The proceedings of the thirtieth SIGCSE technical symposium on Computer science education, p.78-82, March 24-28, 1999, New Orleans, Louisiana, United States
• 30: PMD. 2003. PMD home page. Web page last accessed Mar. 21, 2003. <http://pmd.sourceforge.net/>.
• 31: Kenneth A. Reek, A software infrastructure to support introductory computer science courses, Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education, p.125-129, February 15-17, 1996, Philadelphia, Pennsylvania, United States
• 32: Ricadela, A. 2001. The state of software: Quality. InformationWeek, 838, 43, May 21, 2001.
• 33: Terry Shepard , Margaret Lamb , Diane Kelly, More testing should be taught, Communications of the ACM, v.44 n.6, p.103-108, June 2001 [doi>10.1145/376134.376180]
• 34: Laurie Williams , Richard L. Upchurch, In support of student pair-programming, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.327-331, February 2001, Charlotte, North Carolina, United States
• 35: Rodney C. Wilson, UNIX test tools and benchmarks: methods and tools to design, develop, and execute functional, structural, reliability, and regression tests, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Measurement

Keywords:
agile methods, extreme programming, teaching software testing, test-driven development, test-first coding