Models for Early Identification of Struggling Novice Programmers

ABSTRACT

There is much interest in predicting student performance in computer programming courses early in the semester to identify weak students who might benefit from targeted support. To this end, we analyzed detailed keystroke transcripts and outputs of compilation attempts during programming activities, both in and out of class. In linear regression models predicting grades, we identified multiple behavioral indicators and performance indicators that explained a significant portion of the variation in final grades using only the data collected within the first three weeks. Because the indicators identify specific behaviors and are generated automatically, they may be used as the basis for interventions instructors may use when counseling weaker students concerning their performance early in the course before they fall too far behind. Furthermore, in contrast with some other automated struggling-student detection models, our predictors are based on generic behaviors and generic performance metrics that can be extended to a wide range of introductory programming contexts. Our models also predict performance on a continuous scale rather than a binary "weak"/"not weak" classification, which would allow instructors to offer interventions to marginal students who want to improve, or to promising students who want to excel.

References

1. Alireza Ahadi, Raymond Lister, Heikki Haapala, and Arto Vihavainen. 2015. Exploring machine learning methods to automatically identify students in need of assistance. Proceedings of the Eleventh Annual International Conference on International Computing Education Research. ACM, 121--130.
2. Ali Alammary, Angela Carbone, and Judy Sheard. 2012. Implementation of a smart lab for teachers of novice programmers Proceedings of the Fourteenth Australasian Computing Education Conference-Volume 123. Australian Computer Society, Inc., 121--130.
3. Donald T Campbell and Donald W Fiske. 1959. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin Vol. 56, 2 (1959), 81.
4. Jason Carter, Prasun Dewan, and Mauro Pichiliani. 2015. Towards incremental separation of surmountable and insurmountable programming difficulties. Proceedings of the 46th ACM Technical Symposium on Computer Science Education. ACM, 241--246.
5. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. Understanding the syntax barrier for novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education. ACM, 208--212.
6. John Dunlosky, Katherine A Rawson, Elizabeth J Marsh, Mitchell J Nathan, and Daniel T Willingham. 2013. Improving students' learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, Vol. 14, 1 (2013), 4--58.
7. Anthony Estey, Hieke Keuning, and Yvonne Coady. 2017. Automatically Classifying Students in Need of Support by Detecting Changes in Programming Behaviour. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17). ACM, 189--194.
8. Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, et almbox.. 2015. Educational data mining and learning analytics in programming: Literature review and case studies. Proceedings of the 2015 ITiCSE on Working Group Reports. ACM, 41--63.
9. Matthew C Jadud. 2006. Methods and tools for exploring novice compilation behaviour. Proceedings of the Second International Workshop on Computing Education Research. ACM, 73--84.
10. Charles M Judd, Eliot R Smith, and Louise H Kidder. 1991. Research methods in social relations. (1991).
11. Sean HK Kang. 2016. Spaced Repetition Promotes Efficient and Effective Learning: Policy Implications for Instruction. Policy Insights from the Behavioral and Brain Sciences, Vol. 3, 1 (2016), 12--19.
12. Juho Leinonen, Leo Leppanen, Petri Ihantola, and Arto Hellas. 2017. Comparison of Time Metrics in Programming. In Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM, 200--208.
13. Jonathan P. Munson. 2017. Metrics for timely assessment of novice programmers. Journal of Computing Sciences in Colleges Vol. 32, 3 (January. 2017), 136--148.
14. Jonathan P. Munson and Elizabeth A. Schilling. 2016. Analyzing novice programmers' response to compiler error messages. Journal of Computing Sciences in Colleges Vol. 31, 3 (January. 2016), 53--61.
15. Christian Murphy, Gail Kaiser, Kristin Loveland, and Sahar Hasan. 2009. Retina: helping students and instructors based on observed programming activities. ACM SIGCSE Bulletin, Vol. 41, 1 (2009), 178--182.
16. Ma Mercedes T Rodrigo, Ryan S Baker, Matthew C Jadud, Anna Christine M Amarra, Thomas Dy, Maria Beatriz V Espejo-Lahoz, Sheryl Ann L Lim, Sheila AMS Pascua, Jessica O Sugay, and Emily S Tabanao. 2009. Affective and behavioral predictors of novice programmer achievement ACM SIGCSE Bulletin, Vol. Vol. 41. ACM, 156--160.
17. Emily S Tabanao, Ma Mercedes T Rodrigo, and Matthew C Jadud. 2011. Predicting at-risk novice Java programmers through the analysis of online protocols. Proceedings of the Seventh International Workshop on Computing Education Research. ACM, 85--92.
18. Christopher Watson, Frederick WB Li, and Jamie L Godwin. 2013. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Proceedings of the 2013 IEEE 13th International Conference on Advanced Learning Technologies (ICALT '13). IEEE, 319--323.
19. Christopher Watson, Frederick WB Li, and Jamie L Godwin. 2014. No tests required: comparing traditional and dynamic predictors of programming success. Proceedings of the 45th ACM Technical Symposium on Computer Science Education. ACM, 469--474.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Computer science education