"Whither, badges?" or "wither, badges!": a metastudy of badges in computer science education to clarify effects, significance and influence

ABSTRACT

Gamification is becoming a widely used technique in an attempt to improve engagement and effectiveness of learning, with virtual rewards (often in the form of badges) frequently forming a core part of this structure. But how do badges actually work? Recent studies in gamification have clearly identified that while we can say this approach works, any analysis of the success stories quickly confirms that context is essential in both the system we gamify and the users that take part. In the absence of context, a badge is obviously meaning and valueless, but studies in badge use generally refer to success without focusing on the underlying process and, in some cases, report significance without effect size, report the badge without the process or do not sufficiently isolate the variables to allow us to address what the influence of the badge is. In this paper, we classify the existing work to identify the existing research where badges have been evaluated in a way that appears significant, with properly established control groups, isolation of factors and longitudinal considerations, and with an exposure of the underlying process to determine if what we are seeing is a successful process in action that has a badge at the end, rather than a process that has been made successful through badge use. A badge system can be a proxy for scaffolding and an individual badge can give focus much in the way we would expect in cognitive apprenticeship but both of these outcomes have strong implications that there is a definite limit to how many badges we award, the way in which they are contextualised and the training framework in which they are presented.

References

1. S. Abramovich, C. Schunn, and R. Higashi. Are badges useful in education? it depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61(2):217--232, 2013.
2. B. Alberts. An eduction that inspires. Science, 330(6003):427, 2010.
3. A. Anderson, D. Huttenlocher, J. Klienberg, and J. Leskovec. Engaging with massive online courses. In Proceedings of WWW'2014, pages 687--697, April 2014.
4. J. Antin and E. F. Churchill. Badges in social media: A social psychological perspective. CHI 2011 Gamification Workshop Proceedings (Vancouver, BC, Canada, 2011), 2011.
5. J. C. Burguillo. Using game theory and competition-based learning to stimulate student motivation and performance. Comput. Educ., 55(2):566--575, Sept. 2010.
6. C. Chan, M. Bhandar, L.-B. Oh, and H.-C. Chan. Recognition and participation in a virtual community. In System Sciences, 2004. Proceedings of the 37th Annual Hawaii International Conference on, pages 10 pp.--, Jan 2004.
7. A. Collins, J. S. Brown, and A. Holum. Cognitive apprenticeship: Making thinking visible. American educator, 15(3):6--11, 1991.
8. S. de Sousa Borges, H. Reis, V. Durelli, and S. Isotani. A systematic mapping on gamification applied to education. In Proceedings of SAC'2014, pages 216--222, March 2014.
9. E. Deci, R. Koestner, and R. M. Ryan. A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6):627--668, 1999.
10. S. Deterding, D. Dixon, R. Khaled, and L. Nacke. From game design elements to gamefulness: Defining gamification. In Proceedings of MindTrek'11, pages 9--15, September 2011.
11. A. Elliot. Approach and avoidance motivation and achievement goals. Educational psychologist, 34(3):169--189, 1999.
12. R. Farzan, J. M. DiMicco, D. R. Millen, B. Brownholtz, W. Geyer, and C. Dugan. When the experiment is over: Deploying an incentive system to all the users. 2008.
13. Z. Fitz-Walter, D. Tjondronegoro, and P. Wyeth. Orientation passport: Using gamification to engage university students. In Proceedings of the 23rd Australian Computer-Human Interaction Conference, OzCHI '11, pages 122--125, New York, NY, USA, 2011. ACM.
14. D. Gibson, N. Ostashewski, K. Flintoff, S. Grant, and E. Knight. Digital badges in education. Education and Information Technologies, November 2013.
15. L. Haaranen, P. Ihantola, L. Hakulinen, and A. Korhonen. How (not) to introduce badges to online exercises. In Proceedings of SIGCSE'2014, pages 33--38, March 2014.
16. L. Hakulinen and T. Auvinen. The effect of gamification on students with different achievement goal orientations. In Proceedings of the 2014 Learning and Teaching in Computing and Engineering Conference (Latice 2014), pages 9--16, 2014.
17. L. Hakulinen, T. Auvinen, and A. Korhonen. Empirical study on the effect of achievement badges in TRAKLA2 online learning environment. In Proceedings of the 2013 Learning and Teaching in Computing and Engineering Conference (Latice 2013), pages 47--54, 2013.
18. J. Hamari, J. Koivisto, and H. Sarsa. Does gamification work? - a literature review of empirical studies on gamification. In Proceedings of the 47th Hawaii International Conference on System Sciences, pages 3025--3034, 2014.
19. S. Harter. A new self-report scale of intrinsic versus extrinsic orientation in the classroom: Motivational and informational components. Developmental Psychology, 17(3):300--312, May 1981.
20. J. Husman, W. Pitt Derryberry, H. Michael Crowson, and R. Lomax. Instrumentality, task value, and intrinsic motivation: Making sense of their independent interdependence. Contemporary Educational Psychology, 29(1):63--76, 2004.
21. J. Lee and J. Hammer. Gamification in education: What, how, why bother? Academic Exchange Quarterly, 15(2):147, 2011.
22. H.-F. Lin. Effects of extrinsic and intrinsic motivation on employee knowledge sharing intentions. Journal of information science, 2007.
23. M. Maehr and A. Zusho. Handbook of Motivation in School, chapter Achievement goal theory: The past, present and future, pages 77--104. New York: Routledge, 2009.
24. P. McLaren. Revolutionary pedagogy in post-revolutionary times: Rethinking the political economy of critical education. Educational Theory, 48(4):431--462, 1998.
25. R. B. Miller, T. K. DeBacker, and B. A. Greene. Perceived instrumentality and academics: The link to task valuing. Journal of Instructional Psychology, 1999.
26. B. Morrison and B. DiSalvo. Khan academy gamifies computer science. In Proceedings of SIGCSE'2014, pages 39--44, March 2014.
27. S. Nicholson. A user-centered theoretical framework for meaningful gamification. In Games+Learning+Society 8.0, 2012.
28. M. Piccioni, C. Estler, and B. Meyer. Spoc-supported introduction to programming. In Proceedings of ITiCSE'2014, pages 3--8, June 2014.
29. P. Pintrich. Multiple goals, multiple pathways: The role of goal orientation in learning and achievement. Journal of Educational Psychology, 92(3):544--555, 2000.
30. D. H. Robinson, R. T. Fouladi, N. J. Williams, and S. J. Bera. Some effects of including effect size and "what if" information. The Journal of Experimental Education, 70(4):365--382, 2002.
31. V. K. Singh, R. Jain, and M. S. Kankanhalli. Motivating contributors in social media networks. In Proceedings of the First SIGMM Workshop on Social Media, WSM '09, pages 11--18, New York, NY, USA, 2009. ACM.
32. J. Thom, D. Millen, and J. DiMicco. Removing gamification from an enterprise sns. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW '12, pages 1067--1070, New York, NY, USA, 2012. ACM.
33. T. Urdan and C. Midgley. Academic self-handicapping: What we know, what more there is to learn. Educational Psychology Review, 13(2):115--138, 2001.
34. D. B. Wilson. Practical meta-analysis. 2001.
35. C. Wolters. Advancing achievement goal theory: using goal structures and goal orientations to predict students' motivation, cognition and achievement. Journal of Educational Psychology, 96(2), 2004.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Computer science education