Novice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs

Abstract

Teaching is the process of conveying knowledge and skills to learners. It involves preventing misunderstandings or correcting misconceptions that learners have acquired. Thus, effective teaching relies on solid knowledge of the discipline, but also a good grasp of where learners are likely to trip up or misunderstand. In programming, there is much opportunity for misunderstanding, and the penalties are harsh: failing to produce the correct syntax for a program, for example, can completely prevent any progress in learning how to program. Because programming is inherently computer-based, we have an opportunity to automatically observe programming behaviour -- more closely even than an educator in the room at the time. By observing students’ programming behaviour, and surveying educators, we can ask: do educators have an accurate understanding of the mistakes that students are likely to make? In this study, we combined two years of the Blackbox dataset (with more than 900 thousand users and almost 100 million compilation events) with a survey of 76 educators to investigate which mistakes students make while learning to program Java, and whether the educators could make an accurate estimate of which mistakes were most common. We find that educators’ estimates do not agree with one another or the student data, and discuss the implications of these results.

References

1. Alireza Ahadi, Vahid Behbood, Arto Vihavainen, Julia Prior, and Raymond Lister. 2016. Students’ syntactic mistakes in writing seven different types of SQL queries and its application to predicting students’ success. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE’16). ACM, New York, NY, 401--406.
2. Alireza Ahadi, Raymond Lister, Heikki Haapala, and Arto Vihavainen. 2015. Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the 11th Annual International Conference on International Computing Education Research (ICER’15). ACM, New York, NY, 121--130.
3. Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. 2005. An analysis of patterns of debugging among novice computer science students. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE’05). ACM, New York, NY, 84--88.
4. Amjad Altadmri and Neil C. C. Brown. 2015. 37 million compilations: Investigating novice programming mistakes in large-scale student data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE’15). ACM, New York, NY, 522--527.
5. Yifat Ben-David Kolikant. 2011. Computer science education as a cultural encounter: A socio-cultural framework for articulating teaching difficulties. Instructional Science 39, 4, 543--559.
6. Neil C. C. Brown and Amjad Altadmri. 2014. Investigating novice programming mistakes: Educator beliefs vs. student data. In Proceedings of the 10th Annual Conference on International Computing Education Research (ICER’14). ACM, New York, NY, 43--50.
7. Neil C. C. Brown, Michael Kölling, Davin McCall, and Ian Utting. 2014. Blackbox: A large scale repository of novice programmers’ activity. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE’14). ACM, New York, NY, 223--228.
8. R. Cadenhead. 2012. Sams Teach Yourself Java in 21 Days. Pearson Education.
9. Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2012. All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE’12). ACM, New York, NY, 75--80.
10. Thomas Dy and Ma. Mercedes Rodrigo. 2010. A detector for non-literal Java errors. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling’10). ACM, New York, NY, 118--122.
11. C. Hoisington. 2012. Android Boot Camp for Developers Using Java. Cengage Learning.
12. David C. Howell. 2002. Statistical Methods for Psychology (5th ed.). Duxbury.
13. Maria Hristova, Ananya Misra, Megan Rutter, and Rebecca Mercuri. 2003. Identifying and correcting Java programming errors for introductory computer science students. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’03). ACM, New York, NY, 153--156.
14. Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H. Edwards, Essi Isohanni, et al. 2015. Educational data mining and learning analytics in programming: Literature review and case studies. In Proceedings of the 2015 ITiCSE on Working Group Reports (ITICSE-WGR’15). ACM, New York, NY, 41--63.
15. J. Jackson, M. Cobb, and C. Carver. 2005. Identifying top Java errors for novice programmers. In Proceedings of the 35th Annual Conference on Frontiers in Education (FIE’05).
16. Matthew C. Jadud. 2006. Methods and tools for exploring novice compilation behaviour. In Proceedings of the 2nd International Workshop on Computing Education Research (ICER’06). ACM, New York, NY, 73--84.
17. Matthew C. Jadud and Brian Dorn. 2015. Aggregate compilation behavior: Findings and implications from 27,698 users. In Proceedings of the 11th Annual International Conference on International Computing Education Research (ICER’15). ACM, New York, NY, 131--139.
18. Maurice Kendall. 1990. Rank Correlation Methods (5th ed.). Edward Arnold.
19. Michael Kölling, Bruce Quig, Andrew Patterson, and John Rosenberg. 2003. The BlueJ system and its pedagogy. Computer Science Education 13, 4, 249--268.
20. S. B. Lyerly. 1952. The average Spearman rank correlation coefficient. Psykometrika 17, 421--428.
21. Davin McCall and Michael Kölling. 2014. Meaningful categorisation of novice programmer errors. In Proceedings of the 2014 Frontiers in Education Conference.
22. Andrew Petersen, Jaime Spacco, and Arto Vihavainen. 2015. An exploration of error quotient in multiple contexts. In Proceedings of the 15th Koli Calling Conference on Computing Education Research (Koli Calling’15). ACM, New York, NY, 77--86.
23. Philip M. Sadler, Gerhard Sonnert, Harold P. Coyle, Nancy Cook-Smith, and Jaimie L. Miller. 2013. The influence of teachers’ knowledge on student learning in middle school physical science classrooms. American Educational Research Journal 50, 5, 1020--1049.
24. Elliot Soloway and Sitharama Iyengar (Eds.). 1986. Empirical Studies of Programmers: Papers Presented at the First Workshop on Empirical Studies of Programmers. Intellect Books.
25. Jaime Spacco, Paul Denny, Brad Richards, David Babcock, David Hovemeyer, James Moscola, and Robert Duvall. 2015. Analyzing student work patterns using programming exercise data. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE’15). ACM, New York, NY, 18--23.
26. Jaime Spacco, Davide Fossati, John Stamper, and Kelly Rivers. 2013. Towards improving programming habits to create better computer science course outcomes. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE’13). ACM, New York, NY, 243--248.
27. James C. Spohrer and Elliot Soloway. 1986. Novice mistakes: Are the folk wisdoms correct? Communications of the ACM 29, 7, 624--632.
28. Emily S. Tabanao, Ma. Mercedes T. Rodrigo, and Matthew C. Jadud. 2011. Predicting at-risk novice Java programmers through the analysis of online protocols. In Proceedings of the 7th International Workshop on Computing Education Research (ICER’11). ACM, New York, NY, 85--92.
29. Wilson L. Taylor. 1964. Correcting the average rank correlation coefficient for ties in rankings. Journal of the American Statistical Association 59, 307, 872--876. http://www.jstor.org/stable/2283105
30. Wilson L. Taylor and Ching Fong. 1963. Some contributions to average rank correlation methods and to the distribution of the average rank correlation coefficient. Journal of the American Statistical Association 58, 303, 756--769. http://www.jstor.org/stable/2282723
31. P. van der Linden. 2004. Just Java 2. Pearson Education.
32. Arto Vihavainen, Juha Helminen, and Petri Ihantola. 2014. How novices tackle their first lines of code in an IDE: Analysis of programming session traces. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research (Koli Calling’14). ACM, New York, NY, 109--116.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Computer science education

        • Information science education