Using Spectrum-Based Fault Location and Heatmaps to Express Debugging Suggestions to Student Programmers

ABSTRACT

Automated grading systems are useful for conveying debugging feedback to students, but the manner in which this feedback is displayed can be problematic for students. Automated systems usually report failures in student code, whether it be the results of test case failures or runtime errors. However, since most students are not explicitly taught how to debug, many get stuck. They know the defects exist, but they don't have to experience to know how to find them. Much work has been done in software engineering research using automatic fault localization to programmatically locate bugs within a piece of code under test, as well as research into validating those detection models when applied to student code.

However, determining how to present that information back to the student has not been addressed. There is a balance between giving hints and telling the student exactly where the error occurred. The goal of this paper is to present a technique to present the student's code with suggestions to the student of where to investigate, based on the results of the automatic fault location. Taking results from the GZoltar statistical fault localization library, we have developed a method of expressing the suggestions to the students by visualizing the potential defect locations within the context of the student's source code using heatmaps.

References

1. Aaltonen, K., Ihantola, P., and Seppälä, O. Mutation analysis vs. code coverage in automated assessment of students' testing skills. In Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion (OOPSLA '10). ACM, New York, NY, USA, 153--160.
2. Abreu, R., Zoeteweij, P., Golsteijn, R., and van Gemund, A.J.C. A practical evaluation of spectrum-based fault localization. Journal of Systems and Software 82, 11 (2009), 1780--1792.
3. Astels, D. Test Driven Development: A Practical Guide. Prentice Hall, 2003.
4. Beck, K. Test-Driven Development: By Example. Addison-Wesley, 2003.
5. Buckland, M. and Gey, F. The relationship between Recall and Precision. Journal of the American Society for Information Science 45, 1 (1994), 12--19.
6. Campos, J., Riboira, A., Perez, A., and Abreu, R. GZoltar: an eclipse plug-in for testing and debugging. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE 2012). ACM, New York, NY, USA, 378--381.
7. Cleverdon, C.W. On the Inverse Relationship of Recall and Precision. Journal of Documentation 28, 3 (1972), 195--201.
8. Dallmeier, V., Lindig, C., and Zeller, A. Lightweight Defect Localization for Java. In ECOOP 2005 - Object-Oriented Programming. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005, 528--550.
9. Edmison, B. and Edwards, S.H. Applying spectrum-based fault localization to generate debugging suggestions for student programmers. Software Reliability Engineering Workshops (ISSREW), 2015 IEEE International Symposium on, Gaithersburg, MD, 2015, pp. 93--99.
10. Edwards, S.H. Web-CAT. web-cat.cs.vt.edu, 2016.
11. Edwards, S.H. and Pérez-Quiñones, M.A. Experiences using test-driven development with an automated grader. Journal of Computing Sciences in Colleges 22, 3 (2007), 44--50.
12. Edwards, S.H. and Shams, Z. Do student programmers all tend to write the same software tests? In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITiCSE '14). ACM, New York, NY, USA, 171--176.
13. Gerdes, A., Jeuring, J., and Heeren, B. An interactive functional programming tutor. In Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education (ITiCSE '12). ACM, New York, NY, USA, 250--255.
14. Harrold, M.J., Rothermel, G., Sayre, K., and Wu, R. An empirical investigation of the relationship between spectra differences and regression faults. Journal of Software Testing, Verification and Reliability 10, 3 (2000), 171--194.
15. Hattie, J. and Timperley, H. The power of feedback. Review of Educational Research 77, 1 (2007), 81--112.
16. Joblove, G.H., Greenberg, D., Joblove, G.H., and Greenberg, D. Color spaces for computer graphics. ACM SIGGRAPH Computer Graphics 12, 3 (1978), 20--25.
17. Jones, J.A. and Harrold, M.J. Empirical evaluation of the tarantula automatic fault-localization technique. Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, (ASE '05). ACM, New York, NY, USA, 273--282.
18. Kulhavy, R. Feedback in Written Instruction. Review of Educational Research 47, 2 (1977), 211--232.
19. Lewis, C. Attitudes and beliefs about computer science among students and faculty. ACM SIGCSE Bulletin, 39, 2 (June 2007), 37--41.
20. Long, M.H. Linguistic and conversational adjustments to non-native speakers. Studies in second language acquisition 5, 2 (1983) 177--193.
21. Mackey, A. Feedback, noticing and instructed second language learning. Applied linguistics 27, 3 (2006) 405--430.
22. Pears, A., Seidman, S., Malmi, L., et al. A survey of literature on the teaching of introductory programming. In Working group reports on ITiCSE on Innovation and technology in computer science education (ITiCSE-WGR '07), Janet Carter and June Amillo (Eds.). ACM, New York, NY, USA, 204--223.
23. Shah, A. Web-CAT: A Web-based Center for Automated Testing. (2003).
24. Shams, Z. and Edwards, S.H. Toward practical mutation analysis for evaluating the quality of student-written software tests. In Proceedings of the ninth annual international ACM conference on International computing education research (ICER '13). ACM, New York, NY, USA, 53--58.
25. Wang, S. and Wu, P. The role of feedback and self-efficacy on web-based learning: The social cognitive perspective. Computers & Education 51, 4 (2008), 1589--1598.