License to evaluate: preparing learning analytics dashboards for educational practice

ABSTRACT

Learning analytics can bridge the gap between learning sciences and data analytics, leveraging the expertise of both fields in exploring the vast amount of data generated in online learning environments. A typical learning analytics intervention is the learning dashboard, a visualisation tool built with the purpose of empowering teachers and learners to make informed decisions about the learning process. Related work has investigated learning dashboards, yet none have explored the theoretical foundation that should inform the design and evaluation of such interventions. In this systematic literature review, we analyse the extent to which theories and models from learning sciences have been integrated into the development of learning dashboards aimed at learners. Our analysis revealed that very few dashboard evaluations take into account the educational concepts that were used as a theoretical foundation for their design. Furthermore, we report findings suggesting that comparison with peers, a common reference frame for contextualising information on learning analytics dashboards, was not perceived positively by all learners. We summarise the insights gathered through our literature review in a set of recommendations for the design and evaluation of learning analytics dashboards for learners.

References

1. Sema Alay and Settar Kocak. 2002. Validity and reliability of time management questionnaire. Hacettepe Universitesi Egitim Fakultesi Derg 22 (2002), 9--13.
2. Liaqat Ali, Mohsen Asadi, Dragan Gašević, Jelena Jovanović, and Marek Hatala. 2013. Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education 62 (2013), 130--148.
3. Naif Radi Aljohani and Hugh C Davis. 2013. Learning analytics and formative assessment to provide immediate detailed feedback using a student centered mobile dashboard. In Next Generation Mobile Apps, Services and Technologies (NGMAST), 2013 Seventh International Conference on. IEEE, 262--267.
4. Antonio R Anaya, Manuel Luque, and Manuel Peinado. 2016. A visual recommender tool in a collaborative learning experience. Expert Systems with Applications 45 (2016), 248--259.
5. Ryan S Baker. 2016. Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education 26, 2 (2016), 600--614.
6. Lucy Barnard, William Y Lan, Yen M To, Valerie Osland Paton, and Shu-Ling Lai. 2009. Measuring self-regulation in online and blended learning environments. The Internet and Higher Education 12, 1 (2009), 1--6.
7. Sanam Shirazi Beheshitha, Marek Hatala, Dragan Gašević, and Srećko Joksimović. 2016. The role of achievement goal orientations when studying effect of learning analytics visualizations. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 54--63.
8. John Biggs, David Kember, and Doris YP Leung. 2001. The revised two-factor study process questionnaire: R-SPQ-2F. British journal of educational psychology 71, 1 (2001), 133--149.
9. Robert Bodily and Katrien Verbert. 2017. Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies (2017).
10. Robert Bodily and Katrien Verbert. 2017. Trends and issues in student-facing learning analytics reporting systems research. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 309--318.
11. John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.
12. Susan Bull, Blandine Ginon, Clelia Boscolo, and Matthew Johnson. 2016. Introduction of learning visualisations and metacognitive support in a persuadable open learner model. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 30--39.
13. Susan Bull, Matthew D Johnson, Carrie Demmans Epp, Drew Masci, Mohammad Alotaibi, and Sylvie Girard. 2014. Formative assessment and meaningful learning analytics. In Advanced Learning Technologies (ICALT), 2014 IEEE 14th International Conference on. IEEE, 327--329.
14. Sven Charleer, Joris Klerkx, Erik Duval, Tinne De Laet, and Katrien Verbert. 2016. Creating effective learning analytics dashboards: Lessons learnt. In European Conference on Technology Enhanced Learning. Springer, 42--56.
15. Chih-Yueh Chou, Shu-Fen Tseng, Wen-Chieh Chih, Zhi-Hong Chen, Po-Yao Chao, K Robert Lai, Chien-Lung Chan, Liang-Chih Yu, and Yi-Lung Lin. 2015. Open Student Models of Core Competencies at the Curriculum Level: Using Learning Analytics for Student Reflection. IEEE Transactions on Emerging Topics in Computing (2015).
16. Linda Corrin and Paula de Barba. 2014. Exploring students' interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ascilite 2014 conference. 629--633.
17. Linda Corrin and Paula de Barba. 2015. How do students interpret feedback delivered via dashboards?. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge. ACM, 430--431.
18. Samantha G Daley, Garron Hillaire, and LeeAnn M Sutherland. 2016. Beyond performance data: Improving student help seeking by collecting and displaying influential data in an online middle-school science curriculum. British Journal of Educational Technology 47, 1 (2016), 121--134.
19. Dan Davis, Ioana Jivet, René F Kizilcec, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. 2017. Follow the successful crowd: raising MOOC completion rates through social comparison at scale. In Proceedings of the 7th International Conference on Learning Analytics and Knowledge.
20. Eva Durall and Begoña Gros. 2014. Learning Analytics as a Metacognitive Tool.. In CSEDU (1). 380--384.
21. Andrew J Elliot, Kou Murayama, and Reinhard Pekrun. 2011. A 3X 2 achievement goal model. Journal of Educational Psychology 103, 3 (2011), 632.
22. Rebecca Ferguson. 2012. Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning 4, 5-6 (2012), 304--317.
23. Leon Festinger. 1954. A theory of social comparison processes. Human relations 7, 2 (1954), 117--140.
24. Stephen Few. 2013. Information Dashboard Design: Displaying data for at-a-glance monitoring. Analytics Press.
25. Dragan Gašević, Shane Dawson, and George Siemens. 2015. Let's not forget: Learning analytics are about learning. TechTrends 59, 1 (2015), 64--71.
26. Sten Govaerts, Katrien Verbert, and Erik Duval. 2011. Evaluating the student activity meter: two case studies. In International Conference on Web-Based Learning. Springer, 188--197.
27. Wolfgang Greller and Hendrik Drachsler. 2012. Translating learning into numbers: A generic framework for learning analytics. Educational technology & society 15, 3 (2012), 42--57.
28. Julio Guerra, Roya Hosseini, Sibel Somyurek, and Peter Brusilovsky. 2016. An intelligent interface for learning content: Combining an open learner model and social comparison to support self-regulated learning and engagement. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 152--163.
29. Marek Hatala, Sanam Shirazi Beheshitha, and Dragan Gasevic. 2016. Associations Between Students' Approaches to Learning and Learning Analytics Visualizations.. In LAL@ LAK. 3--10.
30. Ioana Jivet, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2017. Awareness is not enough: pitfalls of learning analytics dashboards in the educational practice. In European Conference on Technology Enhanced Learning. Springer, 82--96.
31. Jeonghyun Kim, Il-Hyun Jo, and Yeonjeong Park. 2016. Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review 17, 1 (2016), 13--24.
32. Donald L Kirkpatrick. 1977. Evaluating training programs: Evidence vs. proof. Training Dev J (1977).
33. Joris Klerkx, Katrien Verbert, and Erik Duval. 2014. Enhancing learning with visualization techniques. In Handbook of research on educational communications and technology. Springer, 791--807.
34. Elizabeth Koh, Antonette Shibani, Jennifer Pei-Ling Tan, and Helen Hong. 2016. A pedagogical framework for learning analytics in collaborative inquiry tasks: an example from a teamwork competency awareness program. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 74--83.
35. Johannes Konert, Christoph Bohr, Henrik Bellhäuser, and Christoph Rensing. 2016. PeerLA-Assistant for individual learning goals and self-regulation competency improvement in online learning scenarios. In Advanced Learning Technologies (ICALT), 2016 IEEE 16th International Conference on. IEEE, 52--56.
36. Vivekanandan Kumar, David Boulanger, Jeremie Seanosky, Karthikeyan Panneerselvam, Thamarai Selvi Somasundaram, et al. 2014. Competence analytics. Journal of Computers in Education 1, 4 (2014), 251--270.
37. Ming Liu, Rafael A Calvo, Abelardo Pardo, and Andrew Martin. 2015. Measuring and visualizing students' behavioral engagement in writing activities. IEEE Transactions on learning technologies 8, 2 (2015), 215--224.
38. Tomasz D Loboda, Julio Guerra, Roya Hosseini, and Peter Brusilovsky. 2014. Mastery grids: An open source social educational progress visualization. In European Conference on Technology Enhanced Learning. Springer, 235--248.
39. Steven Lonn, Stephen J Aguilar, and Stephanie D Teasley. 2015. Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Computers in Human Behavior 47 (2015), 90--97.
40. Zahia Marzouk, Mladen Rakovic, Amna Liaqat, Jovita Vytasek, Donya Samadi, Jason Stewart-Alonso, Ilana Ram, Sonya Woloshen, Philip H Winne, and John C Nesbit. 2016. What if learning analytics were based on learning science? Australasian Journal of Educational Technology 32, 6 (2016).
41. Lynn McAlpine and Cynthia Weston. 2000. Reflection: Issues related to improving professors' teaching and students' learning. Instructional Science 28, 5 (2000), 363--385.
42. Carolina Mejia, Beatriz Florian, Ravi Vatrapu, Susan Bull, Sergio Gomez, and Ramon Fabregat. 2017. A novel web-based approach for visualization and inspection of reading difficulties on university students. IEEE Transactions on Learning Technologies 10, 1 (2017), 53--67.
43. David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, Prisma Group, et al. 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS med 6, 7 (2009), e1000097.
44. Kasia Muldner, Michael Wixon, Dovan Rai, Winslow Burleson, Beverly Woolf, and Ivon Arroyo. 2015. Exploring the impact of a learning dashboard on student affect. In International Conference on Artificial Intelligence in Education. Springer, 307--317.
45. Kyparisia A Papanikolaou. 2015. Constructing interpretative views of learners' interaction behavior in an open learner model. IEEE Transactions on Learning Technologies 8, 2 (2015), 201--214.
46. Yeonjeong Park and Il-Hyun Jo. 2015. Development of the Learning Analytics Dashboard to Support Students' Learning Performance. J. UCS 21, 1 (2015), 110--133.
47. Paul R Pintrich. 2000. Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of educational psychology 92, 3 (2000), 544.
48. Paul R Pintrich et al. 1991. A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). (1991).
49. José A Ruipérez-Valiente, Pedro J Muñoz-Merino, Derick Leony, and Carlos Delgado Kloos. 2015. ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior 47 (2015), 139--148.
50. Samara Ruiz, Sven Charleer, Maite Urretavizcaya, Joris Klerkx, Isabel Fernández-Castro, and Erik Duval. 2016. Supporting learning by considering emotions: tracking and visualization a case study. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 254--263.
51. Maren Scheffel, Hendrik Drachsler, Karel Kreijns, Joop de Kraker, and Marcus Specht. 2017. Widget, widget as you lead, I am performing well indeed!: using results from an exploratory offline study to inform an empirical online study about a learning analytics widget in a collaborative learning environment. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 289--298.
52. Maren Scheffel, Hendrik Drachsler, Christian Toisoul, Stefaan Ternier, and Marcus Specht. 2017. The proof of the pudding: examining validity and reliability of the evaluation framework for learning analytics. In European Conference on Technology Enhanced Learning. Springer, 194--208.
53. Beat Schwendimann, Maria Rodriguez-Triana, Andrii Vozniuk, Luis Prieto, Mina Boroujeni, Adrian Holzer, Denis Gillet, and Pierre Dillenbourg. 2016. Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies (2016).
54. Simon Buckingham Shum and Ruth Deakin Crick. 2012. Learning dispositions and transferable competencies: pedagogy, modelling and learning analytics. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. ACM, 92--101.
55. George Siemens and Dragan Gasevic. 2012. Guest editorial-Learning and knowledge analytics. Educational Technology & Society 15, 3 (2012), 1--2.
56. Dan Suthers and Katrien Verbert. 2013. Learning analytics as a middle space. In Proceedings of the Third International Conference on Learning Analytics and Knowledge. ACM, 1--4.
57. Bernardo Tabuenca, Marco Kalz, Hendrik Drachsler, and Marcus Specht. 2015. Time will tell: The role of mobile learning analytics in self-regulated learning. Computers & Education 89 (2015), 53--74.
58. Jennifer Pei-Ling Tan, Simon Yang, Elizabeth Koh, and Christin Jonathan. 2016. Fostering 21st century literacies through a collaborative critical reading and learning analytics environment: user-perceived benefits and problematics. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM, 430--434.
59. Dirk T Tempelaar, Alexandra Niculescu, Bart Rienties, Wim H Gijselaers, and Bas Giesbers. 2012. How achievement emotions impact students' decisions for online learning, and what precedes those emotions. The Internet and Higher Education 15, 3 (2012),161--169.
60. Rémi Venant, Philippe Vidal, and Julien Broisin. 2016. Evaluation of Learner Performance During Practical Activities: An Experimentation in Computer Education. In Advanced Learning Technologies (ICALT), 2016 IEEE 16th International Conference on. IEEE, 237--241.
61. Katrien Verbert, Erik Duval, Joris Klerkx, Sten Govaerts, and José Luis Santos. 2013. Learning analytics dashboard applications. American Behavioral Scientist 57, 10 (2013), 1500--1509.
62. Katrien Verbert, Sten Govaerts, Erik Duval, Jose Luis Santos, Frans Van Assche, Gonzalo Parra, and Joris Klerkx. 2014. Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 18, 6 (2014), 1499--1514.
63. Alyssa Wise, Yuting Zhao, and Simone Hausknecht. 2014. Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics 1, 2 (2014), 48--71.
64. Alyssa Friend Wise. 2014. Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge. ACM, 203--211.
65. W-H Wu, H-C Hsiao, P-L Wu, C-H Lin, and S-H Huang. 2012. Investigating the learning-theory foundations of game-based learning: a meta-analysis. Journal of Computer Assisted Learning 28, 3 (2012),265--279.
66. Yesom Yoo, Hyeyun Lee, Il-Hyun Jo, and Yeonjeong Park. 2015. Educational dashboards for smart learning: Review of case studies. In Emerging Issues in Smart Learning. Springer, 145--155.
67. Barry J Zimmerman. 1990. Self-regulated learning and academic achievement: An overview. Educational psychologist 25, 1 (1990), 3--17.
68. Barry J Zimmerman, M Boekarts, PR Pintrich, and M Zeidner. 2000. A social cognitive perspective. Handbook of self-regulation 13, 1 (2000), 695--716.

Index Terms

• Applied computing

  • Education

    • E-learning

• Human-centered computing

  • Visualization

    • Visualization application domains

      • Information visualization

• Information systems

  • Information systems applications

    • Decision support systems

      • Data analytics