Lessons Learned from Testing a Children's Educational Game through Web Deployment

ABSTRACT

Beanstalk is an educational game that teaches balance-fulcrum principles, targeting children ages 5-11. Four versions of the game were deployed on the web through the educational portal Learning.com. Two of the versions incorporate non-player characters that offer opportunities for socio-emotional learning (SEL). Two of the versions include a scientific process of "predict-observe-hypothesize-explain" (POHE), which in effect is in-game testing. This paper reports on a first look into the game logs collected from child players via the portal, allowing for a 2x2 SEL crossed with POHE analysis. Lessons learned from testing variants of an educational game are shared, as well as conclusions drawn from the logs. Surprisingly, the version with both SEL and POHE caused children to stop playing the game sooner. The POHE performance was weakest for the players in the SEL treatment. Rather than help with the science educational goals, SEL may have diminished the experience.

References

1. Christel, M., et al. 2013. Beanstalk: A Unity Game Addressing Balance Principles, Socio-Emotional Learning and Scientific Inquiry. In Proc. Int'l Games Innovation Conf. IEEE, NY, 36--39. DOI: 10.1109/IGIC.2013.6659126.
2. Csikszentmihalyi, M. 1990. Flow: The Psychology of Optimal Experience. Harper and Row, New York, NY.
3. Jansen, B. R. J., and van der Maas, H. L. J. 2002. The Development of Children's Rule Use on the Balance Scale Task. J. Experimental Child Psychology, 81, 383--416.
4. Lomas, D., Patel, K., Forlizzi, J. L., and Koedinger, K. R. 2013. Optimizing challenge in an educational game using large-scale design experiments. In Proc. Conf. on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 89--98.
5. National Research Council. 2012. NRC Committee on a Conceptual Framework for New K-12 Science Education Standards, A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press, Washington, DC.
6. Schell, J. 2008. The Art of Game Design: A Book of Lenses. Morgan Kaufmann, Burlington, MA.
7. Shute, V. J., and Ventura, M. 2013. Stealth Assessment: Measuring and Supporting Learning in Video Games. MIT Press, Cambridge, MA.
8. Siegler, R. S. 1976. Three aspects of cognitive development. Cognitive Psychology 8 (1976), 481--520.
9. Siegler, R. S. and Chen, Z. 2002. Development of Rules and Strategies: Balancing the Old and the New. J. Experimental Child Psychology 81 (2002), 446--457.

Index Terms

• Applied computing

  • Education

    • Computer-assisted instruction

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods