skip to main content
10.1145/2522848.2522896acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

A dynamic multimodal approach for assessing learners' interaction experience

Published: 09 December 2013 Publication History

Abstract

In this paper we seek to model the users' experience within an interactive learning environment. More precisely, we are interested in assessing three extreme trends in the interaction experience, namely flow (a perfect immersion within the task), stuck (a difficulty to maintain focused attention) and off-task (a drop out from the task). We propose a hierarchical probabilistic framework using a dynamic Bayesian network to simultaneously assess the probability of experiencing each trend, as well as the emotional responses occurring subsequently. The framework combines three-modality diagnostic variables that sense the learner's experience including physiology, behavior and performance, predictive variables that represent the current context and the learner's profile, and a dynamic structure that tracks the temporal evolution of the learner's experience. We describe the experimental study conducted to validate our approach. A protocol was established to elicit the three target trends as 44 participants interacted with three learning environments involving different cognitive tasks. Physiological activities (electroencephalography, skin conductance and blood volume pulse), patterns of the interaction, and performance during the task were recorded. We demonstrate that the proposed framework outperforms conventional non-dynamic modeling approaches such as static Bayesian networks, as well as three non-hierarchical formalisms including naive Bayes classifiers, decision trees and support vector machines.

References

[1]
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V., Olmstead, R. E., Tremoulet, P. D., and Craven, P. L. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. AVIAT SPACE ENVIR MD 78, B231-B244.
[2]
Brave, S., and Nass, C. 2002. Emotion in human-computer interaction. In Handbook of human-computer interaction, J. Jacko, and A. Sears, Eds. (Elsevier Science Pub Co.).
[3]
Picard, R. 1997. Affective computing (MIT Press).
[4]
Damasio, A. 1994. Descartes' error: Emotion, reason and the human brain ( Grosset and Putnam, New York).
[5]
Baker, R. S., Corbett, A. T., Roll, I., and Koedinger, K. R. 2008. Developing a generalizable detector of when students game the system. User Model User-Adap 18, 287--314.
[6]
Burleson, W. 2006. Affective learning companions: Strategies for empathetic agents with real-time multimodal affective sensing to foster meta-cognitive approaches to learning, motivation, and perseverance. MIT PhD thesis.
[7]
Conati, C., and Maclaren, H. 2009. Empirically building and evaluating a probabilistic model of user affect. User Model User-Adap 19, 267--303.
[8]
Kapoor, A., Burleson, W., and Picard, R. W. 2007. Automatic prediction of frustration. International Journal of Human-Computer Studies 65, 724--736.
[9]
Fairclough, S. H. 2009. Fundamentals of physiological computing. Interacting with Computers 21, 133--145.
[10]
Craig, S., Graesser, A., Sullins, J., and Gholson, B. 2004. Affect and learning: An exploratory look into the role of affect in learning with autotutor. J EducMedia 29, 241--250.
[11]
Picard, R. W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., Machover, T., Resnick, M., Roy, D., and Strohecker, C. 2004. Affective learning - a manifesto. BT TECHNOL J 22, 253--269.
[12]
Graesser, A. C., D'mello, S. K., Chipman, P., King, B., and McDaniel, B. 2007. Exploring relationships between affect and learning with autotutor. In Proc Int Conf AIED.
[13]
O'Regan, K. 2003. Emotion and e-learning. Journal of Asynchronous Learning Networks 7, 78--92.
[14]
Baker, R. S. J. d., D'Mello, S. K., Rodrigo, M. M. T., and Graesser, A. C. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive--affective states during interactions with three different computer-based learning environments. INT J HUM-COMPUT ST 68, 223--241.
[15]
Muse, L. A., Harris, S. G., and Feild, H. S. 2003. Has the inverted-u theory of stress and job performance had a fair test? Human Performance 16, 349--364.
[16]
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., and Baggett, W. B. 2003. Why do only some events cause learning during human tutoring? Cognition and Instruction 21, 209--249.
[17]
Russell, J. A. 2003. Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145--172.
[18]
Pantic, M., and Rothkrantz, L. J. M. 2004. Facial action recognition for facial expression analysis from static face images. IEEE T SYST MAN CY B 34, 1449--1461.
[19]
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., and Solomon, P. E. 2009. The painful face -- pain expression recognition using active appearance models. Image and Vision Computing 27, 1788--1796.
[20]
Litman, D. J., and Forbes-Riley, K. 2004. Predicting student emotions in computer-human tutoring dialogues. In Proc of the 42nd Annual Meeting on Association for Computational Linguistics (Barcelona, Spain).
[21]
Ji, Q., Lan, P., and Looney, C. 2006. A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE T SYST MAN CY A 36, 862--875.
[22]
Liao, W., Zhang, W., Zhu, Z., and Ji, Q. 2005. A decision theoretic model for stress recognition and user assistance. In Proc Conf AAAI Artif Intell.
[23]
Csikszentmihalyi, M. 1990. The psychology of optimal experience (Harper & Row, New York).
[24]
D'Mello, S., and Graesser, A. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 145--157.
[25]
Murphy, K. P. 2002. Dynamic bayesian networks: Representation, inference and learning. PhD thesis. University of California, Berkeley, CA. USA.
[26]
Lang, P. J. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50, 372--385.
[27]
Pope, A. T., Bogart, E. H., and Bartolome, D. S. 1995. Biocybernetic system evaluates indices of operator engagement in automated task. BIOL PSYCHOL 40, 187--195.
[28]
Chaouachi, M., Chalfoun, P., Jraidi, I., and Frasson, C. 2010. Affect and mental engagement: Towards adaptability for intelligent systems. In Proc Int FLAIRS Conf.
[29]
Jasper, H. H. 1958. The ten-twenty electrode system of the international federation. Electroencephalography and Clinical Neurophysiology 371--375.
[30]
John, O. P., Naumann, L. P., and Soto, C. J. 2008. Paradigm shift to the integrative big-five trait taxonomy: History, measurement, and conceptual issues. In Handbook of personality: Theory and research, O. P. John, R. W. Robins, and L. A. Pervin, Eds.
[31]
Lauritzen, S. L. 1995. The EM algorithm for graphical association models with missing data. Computational Statistics & Data Analysis 19, 191--201.

Cited By

View all
  • (2024)Exploring Computational Thinking Perspectives in Black Communities with Physiological Computing2024 Black Issues in Computing Education (BICE)10.1109/BICE60192.2024.00013(27-32)Online publication date: 1-Feb-2024
  • (2024)Exploring Central-Peripheral Nervous System Interaction Through Multimodal Biosignals: A Systematic ReviewIEEE Access10.1109/ACCESS.2024.339403612(60347-60368)Online publication date: 2024
  • (2024)Artificial intelligence based cognitive state prediction in an e-learning environment using multimodal dataMultimedia Tools and Applications10.1007/s11042-023-18021-xOnline publication date: 16-Jan-2024
  • Show More Cited By

Index Terms

  1. A dynamic multimodal approach for assessing learners' interaction experience

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction
    December 2013
    630 pages
    ISBN:9781450321297
    DOI:10.1145/2522848
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 December 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. biofeedback sensing
    2. dynamic bayesian networks
    3. emotional responses
    4. flow
    5. interaction experience
    6. off-task
    7. stuck

    Qualifiers

    • Research-article

    Conference

    ICMI '13
    Sponsor:

    Acceptance Rates

    ICMI '13 Paper Acceptance Rate 49 of 133 submissions, 37%;
    Overall Acceptance Rate 453 of 1,080 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Exploring Computational Thinking Perspectives in Black Communities with Physiological Computing2024 Black Issues in Computing Education (BICE)10.1109/BICE60192.2024.00013(27-32)Online publication date: 1-Feb-2024
    • (2024)Exploring Central-Peripheral Nervous System Interaction Through Multimodal Biosignals: A Systematic ReviewIEEE Access10.1109/ACCESS.2024.339403612(60347-60368)Online publication date: 2024
    • (2024)Artificial intelligence based cognitive state prediction in an e-learning environment using multimodal dataMultimedia Tools and Applications10.1007/s11042-023-18021-xOnline publication date: 16-Jan-2024
    • (2022)Understanding Clinical Reasoning through Visual Scanpath and Brain Activity AnalysisComputation10.3390/computation1008013010:8(130)Online publication date: 28-Jul-2022
    • (2021)Scaling Depression Level Through Facial Image Processing and Social Media AnalysisCommunication and Intelligent Systems10.1007/978-981-16-1089-9_71(921-933)Online publication date: 29-Jun-2021
    • (2019)Enhancing the Learning Experience Using Real-Time Cognitive EvaluationInternational Journal of Information and Education Technology10.18178/ijiet.2019.9.10.12879:10(678-688)Online publication date: 2019
    • (2018)Local Sequence Alignment for Scan Path Similarity AssessmentInternational Journal of Information and Education Technology10.18178/ijiet.2018.8.7.10868:7(482-490)Online publication date: 2018
    • (2018)Towards Attentive Speed Reading on Small Screen Wearable DevicesProceedings of the 20th ACM International Conference on Multimodal Interaction10.1145/3242969.3243009(278-287)Online publication date: 2-Oct-2018
    • (2018)Profiling Target and Potential Learners Today and into the FutureDesigning Instruction For Open Sharing10.1007/978-3-030-02713-1_2(61-121)Online publication date: 18-Dec-2018
    • (2017)Emotions and personality traits in argumentation: An empirical evaluationArgument & Computation10.3233/AAC-1700158:1(61-87)Online publication date: 10-Mar-2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media