Yun-En Liu
ABSTRACT
We present a general automatic experimentation and hypothesis generation framework that utilizes a large set of users to explore the effects of different parts of an intervention parameter space on any objective function. We also incorporate importance sampling, allowing us to run these automatic experiments even if we cannot give out the exact intervention distributions that we want. To show the utility of this framework, we present an implementation in the domain of fractions and numberlines, using an online educational game as the source of players. Our system is able to automatically explore the parameter space and generate hypotheses about what types of numberlines lead to maximal short-term transfer; testing on a separate dataset shows the most promising hypotheses are valid. We briefly discuss our results in the context of the wider educational literature, showing that one of our results is not explained by current research on multiple fraction representations, thus proving our ability to generate potentially interesting hypotheses to test.
Supplemental Material
- Ainsworth, S. E., Bibby, P. A., and Wood, D. J. Analysing the costs and benefits of multi-representational learning environments. Learning with multiple representations (1998), 120--134.Google Scholar
- Andersen, E., Liu, Y.-E., Snider, R., Szeto, R., Cooper, S., and Popović, Z. On the harmfulness of secondary game objectives. In FDG (2011). Google ScholarDigital Library
- Andersen, E., Liu, Y.-E., Snider, R., Szeto, R., and Popović, Z. Placing a value on aesthetics in online casual games. In CHI (2011). Google ScholarDigital Library
- Andersen, E., O'Rourke, E., Liu, Y.-E., Snider, R., Lowdermilk, J., Truong, D., Cooper, S., and Popović, Z. The impact of tutorials on games of varying complexity. In CHI (2012). Google ScholarDigital Library
- Anderson, J. R., Corbett, A. T., Koedinger, K. R., and Pelletier, R. Cognitive tutors: Lessons learned. The Journal of the Learning Sciences 4, 2 (1995), 167--207.Google ScholarCross Ref
- Ansari, D. Effects of development and enculturation on number representation in the brain. Nature Reviews Neuroscience 9, 4 (2008), 278--291.Google ScholarCross Ref
- Arbreton, A. Student goal orientation and help-seeking strategy use.Google Scholar
- Berman, F., Fox, G., and Hey, A. J. Grid computing: making the global infrastructure a reality, vol. 2. Wiley. com, 2003. Google ScholarDigital Library
- BrainPOP. http://www.brainpop.com/.Google Scholar
- Buhrmester, M., Kwang, T., and Gosling, S. D. Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data Perspectives on Psychological Science 6, 1 (2011), 3--5.Google Scholar
- Churchill, Gilbert A., J. A paradigm for developing better measures of marketing constructs. Journal of Marketing Research 16, 1 (1979), pp. 64--73.Google ScholarCross Ref
- Efron, B., and Tibshirani, R. An introduction to the bootstrap, vol. 57. CRC press, 1993.Google Scholar
- Gagatsis, A., and Elia, I. The effects of different modes of representations on mathematical problem solving. In IGPME, vol. 2 (2004), 447--454.Google Scholar
- Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 1 (1970), 97--109.Google ScholarCross Ref
- Henrich, J., Heine, S. J., and Norenzayan, A. The weirdest people in the world. Behavioral and Brain Sciences 33, 2--3 (2010), 61--83.Google ScholarCross Ref
- Hume, G., Michael, J., Rovick, A., and Evens, M. Hinting as a tactic in one-on-one tutoring. Journal of the Learning Sciences 5, 1 (1996), 23--49.Google ScholarCross Ref
- King, R. D., Rowland, J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., Liakata, M., Markham, M., Pir, P., Soldatova, L. N., Sparkes, A., Whelan, K. E., and Clare, A. The automation of science. Science 324, 5923 (2009), 85--89.Google ScholarCross Ref
- Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with mechanical turk. In CHI (2008). Google ScholarDigital Library
- Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Henne, R., Ferres, J. L., and Melamed, T. Online experimentation at microsoft. In Third Workshop on Data Mining Case Studies and Practice Prize (2009).Google Scholar
- Kohavi, R., Henne, R. M., and Sommerfield, D. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In KDD (2007). Google ScholarDigital Library
- Langley, P. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987. Google ScholarDigital Library
- Larson, C. N. Locating proper fractions on number lines: Effect of length and equivalence. School Science and Mathematics 80, 5 (1980), 423--428.Google ScholarCross Ref
- Lee, Y., Buchanan, B., and Aronis, J. Knowledge-based learning in exploratory science: Learning rules to predict rodent carcinogenicity. Machine Learning 30, 2--3 (1998), 217--240. Google ScholarDigital Library
- Lesh, R., Post, T., and Behr, M. Representations and translations among representations in mathematics learning and problem solving. Problems of representation in the teaching and learning of mathematics (1987), 33--40.Google Scholar
- Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, ACM (New York, NY, USA, 2010), 661--670. Google ScholarDigital Library
- Lomas, D., Patel, K., Forlizzi, J. L., and Koedinger, K. R. Optimizing challenge in an educational game using large-scale design experiments. In CHI (2013). Google ScholarDigital Library
- Moss, J., and Case, R. Developing children's understanding of the rational numbers: A new model and an experimental curriculum. Journal for Research in Mathematics Education 30, 2 (1999), pp. 122--147.Google ScholarCross Ref
- Quinlan, J. R. C4. 5: programs for machine learning, vol. 1. Morgan kaufmann, 1993. Google ScholarDigital Library
- Rau, M. A., Aleven, V., and Rummel, N. Intelligent tutoring systems with multiple representations and self-explanation prompts support learning of fractions. In AIED (2009). Google ScholarDigital Library
- Siegler, R. S., Thompson, C. A., and Schneider, M. An integrated theory of whole number and fractions development. Cognitive Psychology 62, 4.Google Scholar
- Sierpinska, A. On understanding the notion of function. The concept of function: Aspects of epistemology and pedagogy 25 (1992), 23--58.Google Scholar
- Singh, K., and Xie, M. Bootstrap: A statistical method, 2008.Google Scholar
- Stamper, J. C., Lomas, D., Ching, D., Ritter, S., Koedinger, K. R., and Steinhart, J. The rise of the super experiment. In EDM (2012), 196--200.Google Scholar
Index Terms
- Towards automatic experimentation of educational knowledge
Recommendations
Optimizing challenge in an educational game using large-scale design experiments
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsOnline games can serve as research instruments to explore the effects of game design elements on motivation and learning. In our research, we manipulated the design of an online math game to investigate the effect of challenge on player motivation and ...
A trace-based framework for analyzing and synthesizing educational progressions
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsA key challenge in teaching a procedural skill is finding an effective progression of example problems that the learner can solve in order to internalize the procedure. In many learning domains, generation of such problems is typically done by hand and ...
Towards understanding balancing in exertion games
ACE '14: Proceedings of the 11th Conference on Advances in Computer Entertainment TechnologyPlaying exertion games with others can be engaging. However, players with different physical skill levels competing against each other can experience reduced engagement because they are either not challenged enough, or challenged too much. Balancing ...
Comments