skip to main content
10.1145/3017680.3017792acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article
Distinguished Paper

Evaluating Neural Networks as a Method for Identifying Students in Need of Assistance

Published:08 March 2017Publication History

ABSTRACT

Course instructors need to be able to identify students in need of assistance as early in the course as possible. Recent work has suggested that machine learning approaches applied to snapshots of small programming exercises may be an effective solution to this problem. However, these results have been obtained using data from a single institution, and prior work using features extracted from student code has been highly sensitive to differences in context. This work provides two contributions: first, a partial reproduction of previously published results, but in a different context, and second, an exploration of the efficacy of neural networks in solving this problem. Our findings confirm the importance of two features (the number of steps required to solve a problem and the correctness of key problems), indicate that machine learning techniques are relatively stable across contexts (both across terms in a single course and across courses), and suggest that neural network based approaches are as effective as the best Bayesian and decision tree methods. Furthermore, neural networks can be tuned to be reliably pessimistic, so they may serve a complementary role in solving the problem of identifying students who need assistance.

References

  1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.Google ScholarGoogle Scholar
  2. A. Ahadi, R. Lister, H. Haapala, and A. Vihavainen. Exploring machine learning methods to automatically identify students in need of assistance. In Proceedings of the eleventh annual International Conference on International Computing Education Research, pages 121--130. ACM, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. B. A. Becker. A new metric to quantify repeated compiler errors for novice programmers. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, pages 296--301. ACM, 2016. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. J. Bennedsen and M. E. Caspersen. Failure rates in introductory programming. ACM SIGCSE Bulletin, 39(2):32--36, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. Bergin and R. Reilly. Predicting introductory programming performance: A multi-institutional multivariate study. Computer Science Education, 16(4):303--323, 2006. Google ScholarGoogle ScholarCross RefCross Ref
  6. J. Bichsel. Analytics in higher education: Benefits, barriers, progress, and recommendations, 2012. Accessed August 24, 2016.Google ScholarGoogle Scholar
  7. A. S. Carter, C. D. Hundhausen, and O. Adesope. The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the eleventh annual International Conference on International Computing Education Research, pages 141--150. ACM, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. A. El Gamal. An educational data mining model for predicting student performance in programming course. International Journal of Computer Applications, 70(17), 2013.Google ScholarGoogle Scholar
  9. A. Elbadrawy, S. Studham, and G. Karypis. Personalized multi-regression models for predicting students performance in course activities. UMN CS, pages 14--011, 2014.Google ScholarGoogle Scholar
  10. M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim. Do we need hundreds of classifiers to solve real world classification problems. J. Mach. Learn. Res, 15(1):3133--3181, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. Fire, G. Katz, Y. Elovici, B. Shapira, and L. Rokach. Predicting student exam's scores by analyzing social network data. In International Conference on Active Media Technology, pages 584--595. Springer, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10--18, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. D. Hovemeyer and J. Spacco. Progsnap specification. Accessed August 26, 2016.Google ScholarGoogle Scholar
  14. P. Ihantola, A. Vihavainen, A. Ahadi, M. Butler, J. Börstler, S. H. Edwards, E. Isohanni, A. Korhonen, A. Petersen, K. Rivers, M. A. Rubio, J. Sheard, B. Skupas, J. Spacco, C. Szabo, and D. Toll. Educational data mining and learning analytics in programming: Literature review and case studies. In Proceedings of the 2015 ITiCSE on Working Group Reports, ITICSE-WGR '15, pages 41--63, New York, NY, USA, 2015. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. M. C. Jadud. Methods and tools for exploring novice compilation behaviour. In Proceedings of the Second International Workshop on Computing Education Research, ICER '06, pages 73--84, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. N. Jaques and J. Nutini. A comparison of random forests and dropout nets for sign language recognition with the kinect.Google ScholarGoogle Scholar
  17. P. Kinnunen and L. Malmi. Why students drop out CS1 course? In Proceedings of the Second International Workshop on Computing Education Research, pages 97--108, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. S. N. Liao, D. Zingaro, M. A. Laurenzano, W. G. Griswold, and L. Porter. Lightweight, early identification of at-risk cs1 students. In Proceedings of the 2016 ACM Conference on International Computing Education Research, ICER '16, pages 123--131, New York, NY, USA, 2016. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. J. Luo, S. E. Sorour, K. Goda, and T. Mine. Predicting student grade based on free-style comments using word2vec and ann by considering prediction results obtained in consecutive lessons. International Educational Data Mining Society, 2015.Google ScholarGoogle Scholar
  20. B. Minaei-Bidgoli and W. F. Punch. Using genetic algorithms for data mining optimization in an educational web-based system. In Genetic and evolutionary computation conference, pages 2252--2263. Springer, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. D. Mullier, D. Moore, and D. Hobbs. A neural-network system for automatically assessing students. In World conference on educational multimedia, hypermedia and telecommunications, pages 1366--1371, 2001.Google ScholarGoogle Scholar
  22. E. Osmanbegović and M. Suljić. Data mining approach for predicting student performance. Economic Review, 10(1), 2012.Google ScholarGoogle Scholar
  23. A. Petersen, M. Craig, J. Campbell, and A. Tafliovich. Revisiting why students drop CS1. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research, Koli Calling '16, pages 71--80, New York, NY, USA, 2016. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. A. Petersen, J. Spacco, and A. Vihavainen. An exploration of error quotient in multiple contexts. In Proceedings of the 15th Koli Calling Conference on Computing Education Research, pages 77--86. ACM, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. D. L. Richmond, D. Kainmueller, M. Yang, E. W. Myers, and C. Rother. Mapping stacked decision forests to deep and sparse convolutional neural networks for semantic segmentation. arXiv preprint arXiv:1507.07583, 2015.Google ScholarGoogle Scholar
  26. M. M. Rodrigo, E. Tabanao, M. B. Lahoz, and M. C. Jadud. Analyzing online protocols to characterize novice java programmers. Philippine Journal of Science, 138(2):177--190, 2009.Google ScholarGoogle Scholar
  27. M. M. T. Rodrigo, R. S. Baker, M. C. Jadud, A. C. M. Amarra, T. Dy, M. B. V. Espejo-Lahoz, S. A. L. Lim, S. A. Pascua, J. O. Sugay, and E. S. Tabanao. Affective and behavioral predictors of novice programmer achievement. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE '09, pages 156--160, New York, NY, USA, 2009. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. E. S. Tabanao, Ma, and M. C. Jadud. Predicting At-risk Novice Java Programmers Through the Analysis of Online Protocols. In Proceedings of the Seventh International Workshop on Computing Education Research, ICER '11, pages 85--92, New York, NY, USA, 2011. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. A. Vihavainen, T. Vikberg, M. Luukkainen, and M. Partel. Scaffolding students' learning using test my code. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education, pages 117--122. ACM, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. C. Watson and F. W. Li. Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education, pages 39--44. ACM, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. C. Watson, F. W. Li, and J. Godwin. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In In Advanced Learning Technologies, pages 31--323. IEEE, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. M. Yorke. Formative assessment and its relevance to retention. Higher Education Research & Development, 20(2):115--126, 2001. Google ScholarGoogle ScholarCross RefCross Ref
  33. D. Zingaro, Y. Cherenkova, O. Karpova, and A. Petersen. Facilitating code-writing in pi classes. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education, SIGCSE '13, pages 585--590, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Evaluating Neural Networks as a Method for Identifying Students in Need of Assistance

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader