Guillaume Marceau
Kathi Fisler
Shriram Krishnamurthi
ABSTRACT
Error messages are one of the most important tools that a language offers its programmers. For novices, this feed-back is especially critical. Error messages typically contain both a textual description of the problem and an indication of where in the code the error occurred. This paper reports on a series of studies that explore beginning students' inter-actions with the vocabulary and source-expression high-lighting in DrRacket. Our findings demonstrate that the error message significantly fail to convey information accurately to students, while also suggesting alternative designs that might address these problems.
- G. Marceau, K. Fisler, and S. Krishnamurthi, "Measuring the Effectiveness of Error Messages Designed for Novice Programmers," Proceedings of the Symposium on Computer Science Education, 2011. Google Scholar
Digital Library
- P.I. Pavlik, N. Presson, G. Dozzi, and B. Macwhinney, "The FaCT (fact and concept) system: A new tool linking cognitive science with educators," Proceedings of the Conference of the Cognitive Science Society, D. McNamara & G. Trafton, eds., 2007, pp. 397--402.Google Scholar
- N. Wirth, "The Programming Language Pascal," Acta Informatica, vol. 1, 1971, pp. 35--63.Google ScholarDigital Library
- S. Bloch, Picturing Programs, College Publications, 2011.Google Scholar
- V.J. Traver, "On compiler error messages: what they say and what they mean," Technical Report, Computer Languages and Systems Department, Jaume-I University, 2010.Google Scholar
- B. Moskal, D. Lurie, and S. Cooper, "Evaluating the effectiveness of a new instructional approach," Proceedings of the Symposium on Computer Science Education, 2004, pp. 75--79. Google ScholarDigital Library
- D. Hagan and S. Markham, "Teaching Java with the BlueJ environment," Proceedings of Australasian Society for Computers in Learning in Tertiary Education Conference, 2000.Google Scholar
- J.S. Rey, From Alice to BlueJ: a transition to Java, Master's thesis, School of Computing, Robert Gordon University, 2009.Google Scholar
- M.M. Ben-Ari, "Compile and Runtime Errors in Java," http://stwww.weizmann.ac.il/g-cs/benari/oop/errors.pdf, accessed June 15, 2010.Google Scholar
- K. Van Haaster and D. Hagan, "Teaching and Learning with BlueJ: an Evaluation of a Pedagogical Tool," Issues in Informing Science and Information Technology, vol. 1, 2004, pp. 455--470.Google ScholarCross Ref
- M. Crestani, "Experience report: growing programming languages for beginning students," Proceedings of the International Conference on Functional Programming, 2010. Google ScholarDigital Library
- A. Pears, S. Seidman, L. Malmi, L. Mannila, E. Adams, J. Bennedsen, M. Devlin, and J. Paterson, "A survey of literature on the teaching of introductory programming," ACM SIGCSE Bulletin, vol. 39, Dec. 2007, pp. 204--223. Google ScholarDigital Library
- P. Gross and K. Powers, "Evaluating assessments of novice programming environments," Proceedings of the International Workshop on Computing Education Research, New York, New York, USA: 2005, pp. 99--110. Google ScholarDigital Library
- B.S. Fagin and L. Merkle, "Quantitative analysis of the effects of robots on introductory Computer Science education," Journal on Educational Resources in Computing, vol. 2, 2002, pp. 1--18. Google ScholarDigital Library
- R.B.-B. Levy, M. Ben-Ari, and P.A. Uronen, "The Jeliot 2000 program animation system," Computers & Education, vol. 40, 2003, pp. 1--15. Google ScholarDigital Library
- T. Flowers, C. Carver, and J. Jackson, "Empowering students and building confidence in novice programmers through Gauntlet," Frontiers in Education, vol. 1, 2004, p. T3H/10 - T3H/13.Google Scholar
- J. Jackson, M. Cobb, and C. Carver, "Identifying Top Java Errors for Novice Programmers," Proceedings of the Frontiers in Education Conference, 2005, p. T4C--24.Google Scholar
- C. Burrell and M. Melchert, "Augmenting compiler error reporting in the Karel++ microworld," Proceedings of the Conference of the National Advisory Committee on Computing Qualifications, 2007, p. 41--46.Google Scholar
- B.S. Lerner, M. Flower, D. Grossman, and C. Chambers, "Searching for type-error messages," Programming language design and implementation, 2007, p. 425. Google ScholarDigital Library
- L. Zolman, "STLFilt: An STL error message decryptor for C++," http://www.bdsoft.com/tools/stlfilt.html, accessed June 10, 2010, 2005.Google Scholar
- J.M. Chabert and T.F. Higginbotham, "An Investigation of Novice Programmer Errors in IBM 370 (OS) Assembly Language," Proceedings of the ACM Southeast Regional Conference, 1976, pp. 319--323. Google ScholarDigital Library
- J.C. Spohrer and E. Soloway, "Novice mistakes: are the folk wisdoms correct?," Communications of the ACM, vol. 29, 1986. Google ScholarDigital Library
- N. Ragonis and M. Ben-Ari, "On understanding the statics and dynamics of object-oriented programs," ACM SIGCSE Bulletin, vol. 37, 2005, pp. 226--230. Google ScholarDigital Library
- M.C. Jadud, "A First Look at Novice Compilation Behaviour Using BlueJ," Computer Science Education, vol. 15, Mar. 2005, p. 25--40.Google ScholarCross Ref
- M.-H. Ng Cheong Vee, K. Mannock, and B. Meyer, "Empirical study of novice errors and error paths in object-oriented programming," Proceedings of the Conference of the Higher Education Academy, Subject Centre for Information and Computer Sciences, 2006, pp. 54--58.Google Scholar
- J. Hage and P.V. Keeken, "Mining Helium programs with Neon," Technical Report, Department of Information and Computing Sciences, Utrecht University, 2007.Google Scholar
- L. Murphy, G. Lewandowski, R. McCauley, B. Simon, L. Thomas, and C. Zander, "Debugging: the good, the bad, and the quirky - a qualitative analysis of novices' strategies," Proceedings of the Symposium on Computer Science Education, 2008, pp. 163--167. Google ScholarDigital Library
- M.C. Jadud, "Methods and tools for exploring novice compilation behaviour," Proceedings of the International Workshop on Computing Education Research, 2006, p. 73--84. Google ScholarDigital Library
- M.F. Köksal, R.E. Baar, and S. Üsküdarl1, "Screen-Replay: A Session Recording and Analysis Tool for DrScheme," Proceedings of the Scheme and Functional Programming Workshop, Technical Report, California Polytechnic State University, CPSLO-CSC-09-03, 2009, pp. 103--110.Google Scholar
- N.J. Coull, SNOOPIE: development of a learning support tool for novice programmers within a conceptual framework, PhD Thesis, School of Computer Science, University Of St. Andrews, 2008.Google Scholar
- H. Chad Lane and K. VanLehn, "Intention-based scoring: An approach to measuring success at solving the composition problem," Proceedings of the Symposium on Computer Science Education, New York, New York, USA: 2005, pp. 373--377. Google ScholarDigital Library
- M.-H. Nienaltowski, M. Pedroni, and B. Meyer, "Compiler Error Messages: What Can Help Novices?," Proceedings of the Technical Symposium on Computer Science Education, 2008, pp. 168--172. Google ScholarDigital Library
Index Terms
- Mind your language: on novices' interactions with error messages
Recommendations
Measuring the effectiveness of error messages designed for novice programmers
SIGCSE '11: Proceedings of the 42nd ACM technical symposium on Computer science educationGood error messages are critical for novice programmers. Re-cognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research ...
Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
ITiCSE-WGR '19: Proceedings of the Working Group Reports on Innovation and Technology in Computer Science EducationDiagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty ...
The Future of the Error Message: Comparing Large Language Models and Novice Programmer Effectiveness in Fixing Errors
SIGCSE 2024: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2Research on enhancing error message presentation is of great interest to teachers and developers alike because improving Integrated Development Environments (IDEs) increases early student retention and efficiency at all levels with more effective ...
Reviews
Keith Harrow
Anyone who has taught an introductory programming course knows that students are often baffled by the error messages they receive from compilers or integrated development environments (IDEs). On the one hand, this statement seems so obvious that it requires no further discussion. On the other hand, as the authors of this paper note, even if we all believe the statement to be true, it is nice to have some formal verification of this phenomenon in terms of human evaluation. Even more useful would be some recommendations on what can be done to help alleviate the situation. This paper focuses on how an error message influences what a student does next. The authors worked with students using the DrRacket programming language at Brown University, Northeastern University, and Worcester Polytechnic Institute. They used a number of investigative techniques: they logged students' programming sessions and were able to see the error message, where the student edited in response, and how much editing the student did before resubmitting; they talked to the students about their debugging; and they conducted tests to see how well the students understood the vocabulary used in the error messages. The results are sobering. First, many students took the error message and highlighting too literally, as if the compiler knew what the mistake was and how to fix it, and was thus giving the student a hint on how to proceed. Students thought the IDE was saying "edit here" and "look inside here for the problem." A number of students debugged by guessing what the mistake was-that is, by simply trying alternatives-without understanding either the underlying problem or how to fix it. Many students did not understand the vocabulary used in the error messages, even though the DrRacket development team had tried to be sensitive to this issue. Perhaps the most useful feature of the paper is a series of recommendations to instructors and those involved with IDEs. First, the error message should not propose a solution-just diagnose the problem. (Of course, this is exactly the opposite of what students want!) The error message should not lead the student, because of misinterpretation, to an incorrect edit. Specific recommendations include: simplify the vocabulary in the error messages (this is especially important for introductory students); be explicit about inconsistencies (for example, if a function use does not match the definition, show both parts and not just one); and help students match terms to code fragments (for example, a function body). Some of the recommendations make more sense in the authors' environment using the IDE that their team created. For example, they recommend that professors consider the specific IDE when designing lectures, and treat error messages and highlighting as integral parts of the course. This may be feasible in their limited student environment, but it is harder to do in a class where students might be using different IDEs. Although the paper focuses on pedagogic IDEs, the authors also discuss related work with other Java and C++ IDEs. To summarize: "This paper describes a series of formal studies on novice students' understanding of vocabulary and source highlighting." It addresses a problem that all programming instructors face, and it suggests some interesting ways to study that problem. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments