Model solutions and properties for diagnosing student programs in Ask-Elle

ABSTRACT

Ask-Elle is an interactive tutor that supports the stepwise development of simple functional programs. Using Ask-Elle students receive feedback about whether or not they are on the right track, they can ask for a hint when they are stuck, and get suggestions about how to refactor their program. Our tutor generates this feedback from model solutions and properties that a solution should satisfy. This paper studies the feasibility of using model solutions together with the desired properties of solutions to analyse the work of a student. It describes an experiment in which we analyse almost 3500 log entries from students using Ask-Elle to solve functional programming exercises, to determine how many of these programs are diagnosed correctly based on model solutions and the desired properties of solutions. Ask-Elle manages to correctly diagnose 82.9% of the student programs. A further analysis of the student programs and the diagnoses shows that adding some reasonable model solutions, properties of model solutions, and general program transformations would increase this percentage to 92.9%.

References

1. Kirsti M Ala-Mutka. 2005. A Survey of Automated Assessment Approaches for Programming Assignments. Computer Science Education 15, 2 (2005), 83--102.
2. Koen Claessen and John Hughes. 2000. QuickCheck: A Light-weight Tool for Random Testing of Haskell Programs. In Proceedings of ICFP 2000: International Conference on Functional Programming. ACM, 268--279.
3. Alex Gerdes, Johan Jeuring, and Bastiaan Heeren. 2010. Using strategies for assessment of programming exercises. In Proceedings of SIGCSE 2010: the 41st ACM technical symposium on Computer science education, Gary Lewandowski, Steven A. Wolfman, Thomas J. Cortina, and Ellen Lowenfeld Walker (Eds.). ACM, 441--445.
4. Alex Gerdes, Johan Jeuring, and Bastiaan Heeren. 2012. An Interactive Functional Programming Tutor. In Proceedings of ITICSE 2012: the 17th Annual Conference on Innovation and Technology in Computer Science Education, T. Lapidot, J. Gal-Ezer, M. E. Caspersen, and O. Hazzan (Eds.). ACM, 250--255.
5. Bastiaan Heeren, Daan Leijen, and Arjan van IJzendoorn. 2003. Helium, for Learning Haskell. In Proceedings of Haskell 2003: the ACM SIGPLAN Workshop on Haskell. ACM, 62--71.
6. Johan Jeuring, Alex Gerdes, and Bastiaan Heeren. 2011. A programming tutor for Haskell. In Proceedings of CEFP 2011: Lecture Notes of the Central European School on Functional Programming, Viktória Zsók, Zoltan Horváth, and Rinus Plasmeijer (Eds.). LNCS, Vol. 7241. Springer, 1--45.
7. Wei Jin, Tiffany Barnes, John C. Stamper, Michael John Eagle, Matt Johnson, and Lorrie Lehmann. 2012. Program Representation for Automatic Hint Generation for a Data-Driven Novice Programming Tutor. In Proceedings of ITS 2012: the 11th International Conference on Intelligent Tutoring Systems, Stefano A. Cerri, William J. Clancey, Giorgos Papadourakis, and Kitty Panourgia (Eds.). LNCS, Vol. 7315. Springer, 304--309.
8. Simon Peyton Jones et al. 2003. Haskell 98, Language and Libraries. The Revised Report. Cambridge University Press.
9. Kelly Rivers and Kenneth R. Koedinger. 2014. Automating Hint Generation with Solution Space Path Construction. In Proceedings of ITS 2014: the 12th International Conference on Intelligent Tutoring Systems, Stefan Trausan-Matu, Kristy Elizabeth Boyer, Martha Crosby, and Kitty Panourgia (Eds.). LNCS, Vol. 8474. Springer, 329--339.
10. Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013. Automated Feedback Generation for Introductory Programming Assignments. In Proceedings of PLDI 2013: the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 15--26.
11. Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4 (2011), 197--221.
12. Jeroen Voeten. 2001. On the Fundamental Limitations of Transformational Design. ACM Transactions on Design Automation of Electronic Systems 6, 4 (2001), 533--552.
13. Songwen Xu and Yam San Chee. 2003. Transformation-Based Diagnosis of Student Programs for Programming Tutoring Systems. IEEE Transactions on Software Engineering 29, 4 (2003), 360--384.

Index Terms

• Applied computing

  • Education

    • Computer-assisted instruction

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Computer science education