Using computational methods to discover student science conceptions in interview data

ABSTRACT

A large body of research in the learning sciences has focused on students' commonsense science knowledge---the everyday knowledge of the natural world that is gained outside of formal instruction. Although researchers studying commonsense science have employed a variety of methods, one-on-one clinical interviews have played a unique and central role. The data that result from these interviews take the form of video recordings, which in turn are often compiled into written transcripts, and coded by human analysts. In my team's work on learning analytics, we draw on this same type of data, but we attempt to automate its analysis. In this paper, I describe the success we have had using extremely simple methods from computational linguistics---methods that are based on rudimentary vector space models and simple clustering algorithms. These automated analyses are employed in an exploratory mode, as a way to discover student conceptions in the data. The aims of this paper are primarily methodological in nature; I will attempt to show that it is possible to use techniques from computational linguistics to analyze data from commonsense science interviews. As a test bed, I draw on transcripts of a corpus of interviews in which 54 middle school students were asked to explain the seasons.

References

1. Duit, R. 2009. Bibliography: Students' and Teachers' Conceptions and Science Education. Leibniz Institute for Science Education, Kiel, Germany.
2. Vosniadou, S. and Brewer, W. F. 1992. Mental models of the earth: A study of conceptual change in childhood. Cognitive Psychol., 24, 4 (Oct. 1992), 535--585.
3. Samarapungavan, A. and Wiers, R. W. 1997. Children's thoughts on the origin of species: A study of explanatory coherence. Cognitive Sci., 21, 2 (Apr.-Jun. 1997), 147--177.
4. Wellman, H. M. and Johnson, C. N. 1982. Children's understanding of food and its functions: A preliminary study of the development of concepts of nutrition. J. Appl. Dev. Psychol., 3 135--148.
5. McCloskey, M. 1983. Naive theories of motion. In Mental Models, D. Gentner and A. Stevens eds. Erlbaum, Hillsdale, NJ, 289--324.
6. Sherin, B. 2001. How students understand physics equations. Cognition Instruct., 19, 4 479--541.
7. Smith, J. P., diSessa, A. A. and Roschelle, J. 1993. Misconceptions Reconceived: A Constructivist Analysis of Knowledge in Transition. J. Learn. Sci., 3, 2 115--163.
8. Salton, G., Wong, A. and Yang, C. S. 1974. A vector space model for automatic indexing. Dept. of Computer Science Cornell University, Ithaca N Y.
9. Deerwester, S., Dumais, S. T., Furnas, G., Landauer, T. and Harshman, R. 1990. Indexing by latent semantic analysis. J. Am. Soc. Inform. Sci., 41, 6 (Sep. 1990), 391--407.
10. Landauer, T., Foltz, P. W. and Laham, D. 1998. An introduction to latent semantic analysis. Discourse Process., 25, 2--3 259--284.
11. Berry, M. W., Dumais, S. T. and O'Brien, G. W. 1995. Using linear algebra for intelligent information retrieval. Siam Rev., 37, 4 (Dec. 1995), 573--595.
12. Haley, D., Thomas, P., De Roeck, A. and Petre, M. 2005. A Research Taxonomy for Latent Semantic Analysis-Based Educational Applications. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (Borovets, Bulgaria, 2005).
13. Landauer, T., Laham, D. and Foltz, P. W. 2003. Automatic essay assessment. Assessment in Education: Principles, Policy and Practice, 10, 3 295--308.
14. Shapiro, A. M. and McNamara, D. S. 2000. The Use of Latent Semantic Analysis as a Tool for the Quantitative Assessment of Understanding and Knowledge. J. Educ. Comput. Res., 22, 1 1--36.
15. Magliano, J. P. and Millis, K. K. 2003. Assessing reading skill with a think-aloud procedure and latent semantic analysis. Cognition Instruct., 21, 3 251--283.
16. Millis, K., Kim, H.-J. J., Todaro, S., Magliano, J. P., Wiemer-Hastings, K. and McNamara, D. S. 2004. Identifying reading strategies using latent semantic analysis: Comparing semantic benchmarks. Behav. Res. Methods, 36, 2 213--221.
17. Wade-Stein, D. and Kintsch, E. 2004. Summary Street: Interactive computer support for writing. Cognition Instruct., 22, 3 333--362.
18. Wiemer-Hastings, P. and Graesser, A. C. 2000. Select-a-Kibitzer: A Computer Tool that Gives Meaningful Feedback on Student Compositions. Interact. Learn. Envir., 8, 2 (Jan 1 2000), 149--169.
19. Graesser, A. C., Lu, S., Jackson, G. T. and Mitchell, H. 2004. AutoTutor: A tutor with dialogue in natural language. Behav. Res. Methods, 36, 2 (May 2004), 180--192.
20. Graesser, A. C., Wiemer-Hastings, P. and Wiemer-Hastings, K. 2000. Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor. Interact. Learn. Envir. (Jan 1 2000), 129--147.
21. Sherin, B., Krakowski, M. and Lee, V. R. 2012. Some assembly required: How scientific explanations are constructed during clinical interviews. J. Res. Sci. Teach., 49, 2 166--198.
22. Atwood, R. K. and Atwood, V. A. 1996. Preservice Elementary Teachers' Conceptions of the Causes of Seasons. J. Res. Sci. Teach., 33, 5 (May 1996), 553--563.
23. Newman, D. and Morrison, D. 1993. The conflict between teaching and scientific sense-making: The case of a curriculum on seasonal change. Interact. Learn. Envir., 3 1--16.
24. Sadler, P. M. 1987. Alternative conceptions in astronomy. In Proceedings of the Second international seminar on Misconception and Educational Strategies in Science and Mathematics (Ithaca, NY, 1987). Cornell University Press.
25. Trumper, R. 2001. A cross-college age study of science and nonscience students' conceptions of basic astronomy. Journal of Science Education and Technology, 10, 2 192--195.
26. Lelliott, A. and Rollnick, M. 2010. Big Ideas: A Review of Astronomy Education Research 1974--2008. Int. J. Sci. Educ., 32, 13 1771--1799. Doi 10.1080/09500690903214546.
27. Foltz, P. W., Kintsch, W. and Landauer, T. 1998. The measurement of textual coherence with latent semantic analysis. Discourse Process., 25, 2--3 285--307.
28. Dam, G. and Kaufmann, S. 2008. Computer assessment of interview data using latent semantic analysis. Behav. Res. Methods., 40, 1 (Feb. 2008), 8--20. Doi 10.3758/Brm.40.1.8.
29. Manning, C. D., Raghavan, P. and Schütze, H. 2008. Introduction to information retrieval. Cambridge University Press, New York.
30. Hofmann, T. 2001. Unsupervised Learning by Probabilistic Latent Semantic Analysis, 42, 1 177.
31. Blei, D. M., Ng, A. Y. and Jordan, M. I. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res., 3, 4/5 (May 15 2003), 993--1022.

Index Terms

• Applied computing

  • Education

    • Computer-assisted instruction

  • Enterprise computing

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Language resources

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Clustering and classification

  • Information systems applications

    • Data mining

      • Clustering