Connecting Instructors and Learning Scientists via Collaborative Dynamic Experimentation

ABSTRACT

The shift to digital educational resources provides new opportunities to advance psychology and education research, in tandem with improving instruction using theory and data. To realize this potential, this paper explores how randomized experiments can support mutually beneficial instructor-researcher collaborations. We developed the Collaborative Dynamic Experimentation (CDE) framework to address two key tensions. To enable researchers to embed experiments in online lessons while maintaining instructors' editorial control, Collaborative experiment authoring is needed. To enable instructors to use data for rapid improvement while maintaining statistically valid data for researchers, we apply an interpretable machine learning algorithm for Dynamic experimentation. We worked with an on-campus instructor to implement a proof-of-concept CDE system to experiment within their online calculus quizzes. The qualitative results from this deployment provided insight into how the CDE framework can facilitate alignment of research and practice.

References

1. Sasha Barab and Kurt Squire. 2004. Design-based research: Putting a stake in the ground. The journal of the learning sciences 13, 1 (2004), 1--14.
2. Anthony S Bryk, Louis M Gomez, and Alicia Grunow. 2011. Getting ideas into action: Building networked improvement communities in education. In Frontiers in sociology of education. Springer, 127--162.
3. Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. In Advances in neural information processing systems. 2249--2257.
4. Carol S Dweck. 2002. Messages that motivate: How praise molds students' beliefs, motivation, and performance (in surprising ways). (2002).
5. Neil T Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments Ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (2014), 470--497.
6. Kenneth R Koedinger, Ryan SJd Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010. A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining 43 (2010).
7. John Langford and Tong Zhang. 2008. The epochgreedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems. 817--824.
8. J Derek Lomas, Jodi Forlizzi, Nikhil Poonwala, Nirmal Patel, Sharan Shodhan, Kishan Patel, Ken Koedinger, and Emma Brunskill. 2016. Interface Design Optimization as a Multi-Armed Bandit Problem. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4142--4153.
9. Marsha Lovett, Oded Meyer, and Candace Thille. 2008. JIME-The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 2008, 1 (2008).
10. G Mcfall and L Neumann. 2010. IMS learning tools interoperability basic LTI implementation guide v1. 0 Final. Internet: http://www. imsglobal. org/lti/2010 [May 2013] (2010).
11. Tim McNamara and Carsten Roever. 2006. Language testing: The social dimension. Vol. 1. John Wiley & Sons.
12. Jason A Okonofua, David Paunesku, and Gregory M Walton. 2016. Brief intervention to encourage empathic discipline cuts suspension rates in half among adolescents. Proceedings of the National Academy of Sciences (2016), 201523698.
13. Alexander Renkl. 1997. Learning from worked-out examples: A study on individual differences. Cognitive science 21, 1 (1997), 1--29.
14. Alan H Schoenfeld. 1992. Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics. Handbook of research on mathematics teaching and learning (1992), 334--370.
15. Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press.
16. Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. Vol. 1. MIT press Cambridge.
17. John Sweller. 2006. The worked example effect and human cognition. Learning and Instruction 16, 2 (2006), 165--169.
18. Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. In Proceedings of the Third (2016) ACM Conference on Learning at Scale. ACM, 379--388.

Index Terms

• Human-centered computing