Automatic Teacher Modeling from Live Classroom Audio

ABSTRACT

We investigate automatic analysis of teachers' instructional strategies from audio recordings collected in live classrooms. We collected a data set of teacher audio and human-coded instructional activities (e.g., lecture, question and answer, group work) in 76 middle school literature, language arts, and civics classes from eleven teachers across six schools. We automatically segment teacher audio to analyze speech vs. rest patterns, generate automatic transcripts of the teachers' speech to extract natural language features, and compute low-level acoustic features. We train supervised machine learning models to identify occurrences of five key instructional segments (Question & Answer, Procedures and Directions, Supervised Seatwork, Small Group Work, and Lecture) that collectively comprise 76% of the data. Models are validated independently of teacher in order to increase generalizability to new teachers from the same sample. We were able to identify the five instructional segments above chance levels with F1 scores ranging from 0.64 to 0.78. We discuss key findings in the context of teacher modeling for formative assessment and professional development.

References

1. Alibali, M.W., Nathan, M.J., Wolfgram, M.S., Church, R.B., Jacobs, S.A., Johnson Martinez, C. and Knuth, E.J. 2014. How teachers link ideas in mathematics instruction using speech and gesture: A corpus analysis. Cognition and Instruction. 32, 1 (2014), 65--100.
2. Applebee, A.N., Langer, J.A., Nystrand, M. and Gamoran, A. 2003. Discussion-based approaches to developing understanding: Classroom instruction and student performance in middle and high school English. American Educational Research Journal. 40, 3 (2003), 685--730.
3. Barrón-Cedeño, A., Vila, M., Martí, M.A. and Rosso, P. 2013. Plagiarism meets paraphrasing: Insights for the next generation in automatic plagiarism detection. Computational Linguistics. 39, 4 (2013), 917--947.
4. Blanchard, N., Brady, M., Olney, A.M., Glaus, M., Sun, X., Nystrand, M., Samei, B., Kelly, S. and D'Mello, S. 2015. A Study of Automatic Speech Recognition in Noisy Classroom Environments for Automated Dialog Analysis. Artificial Intelligence in Education. Springer International Publishing. 23--33.
5. Blanchard, N., D'Mello, S., Nystrand, M. and Olney, A.M. 2015. Automatic Classification of Question & Answer Discourse Segments from Teacher's Speech in Classrooms. Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015), International Educational Data Mining Society (2015).
6. Caughlan, S., Juzwik, M.M., Borsheim-Black, C., Kelly, S. and Fine, J.G. 2013. English teacher candidates developing dialogically organized instructional practices. Research in the Teaching of English. 47, 3 (2013), 212.
7. D'Mello, S.K., Olney, A.M., Blanchard, N., Samei, B., Sun, X., Ward, B. and Kelly, S. 2015. Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (2015), 557--566.
8. Drugman, T. and Stylianou, Y. 2014. Maximum voiced frequency estimation: Exploiting amplitude and phase spectra. Signal Processing Letters, IEEE. 21, 10 (2014), 1230--1234.
9. Ford, M., Baer, C., Xu, D., Yapanel, U. and Gray, S. 2008. The LENA language environment analysis system. LENA Foundation Technical Report LTR-03-02.
10. Gates Foundation 2013. Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET project's three-year study-Policy and practitioner brief. Bill & Melinda Gates Foundation Seattle, WA.
11. Goldman, R., Pea, R., Barron, B. and Derry, S.J. 2014. Video research in the learning sciences. Routledge.
12. Graesser, A.C. and McNamara, D.S. 2012. Automated analysis of essays and open-ended verbal responses. APA handbook of research methods in psychology. Washington, DC: American Psychological Association. (2012).
13. Hall, J.K. 2008. Language education and culture. Encyclopedia of language and education. Springer. 45--55.
14. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. and Witten, I.H. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter. 11, 1 (2009), 10--18.
15. Juzwik, M.M., Borsheim-Black, C., Caughlan, S. and Heintz, A. 2013. Inspiring dialogue: Talking to learn in the English classroom. Teachers College Press.
16. Kelly, S. 2007. Classroom discourse and the distribution of student engagement. Social Psychology of Education. 10, 3 (2007), 331--352.
17. Lai, M.K. and McNaughton, S. 2013. Analysis and discussion of classroom and achievement data to raise student achievement. Data-based decision making in education. Springer. 23--47.
18. Lartillot, O., Toiviainen, P. and Eerola, T. 2008. A matlab toolbox for music information retrieval. Data analysis, machine learning and applications. Springer. 261--268.
19. Lewis, D.D. 1998. Naive (Bayes) at forty: The independence assumption in information retrieval. Machine learning: ECML-98. Springer. 4--15.
20. Mu, J., Stegmann, K., Mayfield, E., Rosé, C. and Fischer, F. 2012. The ACODEA framework: Developing segmentation and classification schemes for fully automatic analysis of online discussions. International Journal of Computer-Supported Collaborative Learning. 7, 2 (2012), 285--305.
21. Nystrand, M. 2004. CLASS 4.0 user's manual. The National Research Center on. (2004).
22. Nystrand, M. 2006. Research on the role of classroom discourse as it affects reading comprehension. Research in the Teaching of English. (2006), 392--412.
23. Nystrand, M., Gamoran, A., Kachur, R. and Prendergast, C. 1997. Opening dialogue: Understanding the Dynamics of Language and Learning in the English Classroom. Language and Literacy Series.
24. Nystrand, M., Wu, L.L., Gamoran, A., Zeiser, S. and Long, D.A. 2003. Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse processes. 35, 2 (2003), 135--198.
25. Olney, A., Louwerse, M., Matthews, E., Marineau, J., Hite-Mitchell, H. and Graesser, A. 2003. Utterance classification in AutoTutor. Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing-Volume 2 (2003), 1--8.
26. Rus, V., D'Mello, S., Hu, X. and Graesser, A. 2013. Recent advances in conversational intelligent tutoring systems. AI magazine. 34, 3 (2013), 42--54.
27. Samei, B., Olney, A., Kelly, S., Nystrand, M., D'Mello, S., Blanchard, N., Sun, X., Glaus, M. and Graesser, A. 2014. Domain independent assessment of dialogic properties of classroom discourse. Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014) International Educational Data Mining Society (2014).
28. Samei, B., Olney, A.M., Kelly, S., Nystrand, M., Blanchard, S.D.N. and Graesser, A. Modeling Classroom Discourse: Do Models that Predict Dialogic Instruction Properties Generalize across Populations?
29. Sottilare, R.A., Graesser, A., Hu, X. and Holden, H. 2013. Design Recommendations for Intelligent Tutoring Systems: Volume 1-Learner Modeling. US Army Research Laboratory.
30. Wang, Z., Miller, K. and Cortina, K. 2013. Using the LENA in Teacher Training: Promoting Student Involvement through automated feedback. Unterrichtswissenschaft. 4, (2013), 290--305.
31. Wang, Z., Pan, X., Miller, K.F. and Cortina, K.S. 2014. Automatic classification of activities in classroom discourse. Computers & Education. 78, (2014), 115--123.

Index Terms

• Applied computing

  • Education

    • Computer-managed instruction

    • Learning management systems

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Discourse, dialogue and pragmatics

  • Machine learning

    • Learning paradigms

      • Supervised learning

        • Supervised learning by classification

• Social and professional topics

  • Professional topics

    • Computing education

      • K-12 education