No tests required: comparing traditional and dynamic predictors of programming success

ABSTRACT

Research over the past fifty years into predictors of programming performance has yielded little improvement in the identification of at-risk students. This is possibly because research to date is based upon using static tests, which fail to reflect changes in a student's learning progress over time. In this paper, the effectiveness of 38 traditional predictors of programming performance are compared to 12 new data-driven predictors, that are based upon analyzing directly logged data, describing the programming behavior of students. Whilst few strong correlations were found between the traditional predictors and performance, an abundance of strong significant correlations based upon programming behavior were found. A model based upon two of these metrics (Watwin score and percentage of lab time spent resolving errors) could explain 56.3% of the variance in coursework results. The implication of this study is that a student's programming behavior is one of the strongest indicators of their performance, and future work should continue to explore such predictors in different teaching contexts.

References

1. Bennedsen, J. and Caspersen, M.E. 2007. Failure rates in introductory programming. SIGCSE Bull. 39(2), 32--36.
2. Bergin, S. and Reilly, R. 2005. Programming: factors that influence success. SIGCSE Bull. 37(1), 411--415.
3. Bergin, S., Reilly, R. and Traynor, D. 2005. Examining the role of self-regulated learning on introductory programming performance. In Proc. of the 1st Int. Workshop on Computing Education Research (ICER), 81--86.
4. Bergin, S. and Reilly, R. 2005. The influence of motivation and comfort-level on learning to program. In Proc. of the 17th PPIG Workshop, 293--304.
5. Biamonte, A.J. 1964. Predicting success in programmer training. In Proc. of the 2nd SIGCPR Conference on Computer Personnel Research, 9--12.
6. Byrne, P. and Lyons, G. 2001. The effect of student attributes on success in programming. SIGCSE Bull. 33(3), 49--52.
7. Campbell, V. and Johnstone, M. 2010. The Significance of Learning Style with Respect to Achievement in First Year Programming Students. In Proc. of the 21st Australian Software Engineering Conference (ASWEC), 165--170.
8. Cantwell-Wilson, B. and Shrock, S. 2001. Contributing to success in an introductory computer science course: a study of twelve factors. SIGCSE Bull. 33(1), 184--188.
9. Chamillard A.T. and Karolick, D. 1999. Using learning style data in an introductory computer science course. SIGCSE Bull. 31(1), 291--295.
10. Corman, L.S. 1986. Cognitive style, personality type, and learning ability as factors in predicting the success of the beginning programming student. SIGCSE Bull. 18(4), 80--89.
11. Henry, J.W., Martinko, M.J., and Pierce, M.A. 1994. Attributional style as a predictor of success in a first computer science course. Computers in Human Behavior, 9(4), 341--352.
12. Jadud, M.C. 2006. Methods and tools for exploring novice compilation behaviour. Proc. of the 2nd Int. Workshop on Computing Education Research (ICER), 73--84.
13. Lau, W.W., and Yuen, A.H. 2009. Exploring the effects of gender and learning styles on computer programming performance implications for programming pedagogy. British Journal of Educational Technology, 40(4), 696--712.
14. Lau, W.W., and Yuen, A.H. 2011. Modelling programming performance: Beyond the influence of learner characteristics. Computers & Education, 57(1), 1202--1213.
15. Pintrich, P. R., Smith, D. A., García, T., & McKeachie, W. J. 1993. Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and psychological measurement, 53(3), 801--813.
16. Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V, Lim, S.A.L., Pascua, S.A.M.S., Sugay, J.O., and Tabanao, E.S. 2009. Affective and behavioral predictors of novice programmer achievement. SIGCSE Bull. 41(3), 156--160.
17. Stein, M.V. 2002. Mathematical preparation as a basis for success in CS-II. Computing Sciences in Colleges, 17(4), 28--38.
18. Ventura, P. 2005. Identifying predictors of success for an objects-first CS1. Computer Science Education, 15(3), 223--243.
19. Watson, C., Li, F.W.B., and Lau, R.W.H. 2010. A pedagogical interface for authoring adaptive e-learning courses. In Proc. of the 2nd Int. Workshop on Multimedia Technologies for Distance Learning, 13--18.
20. Watson, C., Li, F.W.B., and Godwin, J.L. 2012. BlueFix: using crowd sourced feedback to support programming students in error diagnosis and repair. In Proc. of 11th Int. Conference on Advances in Web-Based Learning, 228--239.
21. Li, F.W.B., and Watson, C. 2011. Game-based concept visualization for learning programming. In Proc. of the 3rd Int. Workshop on Multimedia Technologies for Distance Learning (MTDL), 37--42.
22. Watson, C., Li, F.W.B., and Lau, R.W.H. 2011. Learning programming languages through corrective feedback and concept visualisation. In Proc. of 10th Int. Conference on Advances in Web-Based Learning (ICWL), 11--20.
23. Watson, C., Li, F.W.B., and Godwin, J.L. 2013. Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior. In Proc. of the 13th IEEE International Conference on Advanced Learning Technologies (ICALT), 319--323.
24. White, G., and Sivitanides, M. 2003. An empirical investigation of the relationship between success in mathematics and visual programming courses. Information Systems Education, 14(4), 409--416.
25. Wiedenbeck, S. 2005. Factors affecting the success of non-majors in learning to program. In Proc. of the 1st Int. Workshop on Computing Education Research (ICER), 13--24.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Computer science education