Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State

ABSTRACT

We present Brain Automated Chorales (BACh), an adaptive brain-computer system that dynamically increases the levels of difficulty in a musical learning task based on pianists' cognitive workload measured by functional near-infrared spectroscopy. As users' cognitive workload fell below a certain threshold, suggesting that they had mastered the material and could handle more cognitive information, BACh automatically increased the difficulty of the learning task. We found that learners played with significantly increased accuracy and speed in the brain-based adaptive task compared to our control condition. Participant feedback indicated that they felt they learned better with BACh and they liked the timings of the level changes. The underlying premise of BACh can be applied to learning situations where a task can be broken down into increasing levels of difficulty.

References

1. Daniel Afergan, Evan M. Peck, Erin T. Solovey, Andrew Jenkins, Samuel W. Hincks, Eli T. Brown, Remco Chang, and Robert J.K. Jacob. 2014a. Dynamic difficulty using brain metrics of workload. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (2014), 3797-3806.
2. Daniel Afergan, Tomoki Shibata, Samuel W. Hincks, Evan M. Peck, Beste F. Yuksel, Remco Chang, and Robert J.K. Jacob. 2014b. Brain-Based Target Expansion. Proc. UIST 2014 (2014).
3. Daniel W. Ash and Dennis H. Holding. 1990. Backward versus forward chaining in the acquisition of a keyboard skill. Human Factors 32 (1990), 139-146.
4. Hasan Ayaz, Patricia A Shewokis, Scott Bunce, Kurtulus Izzetoglu, Ben Willems, and Banu Onaral. 2012. Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 1 (2012), 36-47.
5. Miriam H. Beauchamp, Alain Dagher, John A. D. Aston, and Julien. Doyon. 2003. Dynamic functional changes associated with cognitive skill learning of an adapted version of the Tower of London task. NeuroImage 20 (2003), 1649-1660. DOI: http://dx.doi.org/10.1016/j.neuroimage.2003.07.003
6. Ellen R. Bialo and Jay Sivin-Kachala. 1996. The effectiveness of technology in schools: A summary of recent research. (1996).
7. Roland Brünken, Jan L Plass, and Detlev Leutner. 2003. Direct Measurement of Cognitive Load in Multimedia Learning. Educational Psychologist 38, 1 (2003), 53-61. DOI: http://dx.doi.org/10.1207/S15326985EP3801_7
8. Rafael A. Calvo, Sidney D'Mello, Jonathan Gratch, and Arvid Kappas (Eds). 2014. The Oxford Handbook of Affective Computing. Oxford University Press, 2014.
9. Diane Damos and Christopher D. Wickens. 1980. The identification and transfer of time-sharing skills. Acta Psychologica 46 (1980), 15-39.
10. Ton de Jong. 2010. Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science 38 (2010), 105-134. DOI: http://dx.doi.org/10.1007/s11251-009--9110-0
11. Mark D'Esposito, Bradley R. Postle, and Bart Rypma. 2000. Prefrontal cortical contributions to working memory: evidence from event-related fMRI studies. Experimental brain research 133, 1 (2000), 3-11. http://www.ncbi.nlm.nih.gov/pubmed/10933205
12. Stephen H. Fairclough. 2009. Fundamentals of physiological computing. Interacting with Computers 21, 1--2 (Jan. 2009), 133-145. DOI: http://dx.doi.org/10.1016/j.intcom.2008.10.011
13. Peter T Fox, Marcus E Raichle, Mark A Mintun, and Carmen Dence. 1988. Glucose During Focal Physiologic Neural Activity Nonoxidaive Consumption. Science 241 (1988), 462-464.
14. Tycho K. Fredericks, Sang D. Choi, Jason Hart, Steven E. Butt, and Anil Mital. 2005. An investigation of myocardial aerobic capacity as a measure of both physical and cognitive workloads. International Journal of Industrial Ergonomics 35 (2005), 1097-1107. DOI: http://dx.doi.org/10.1016/j.ergon.2005.06.002
15. Hugh Garavan, Dan Kelley, Allyson Rosen, Steve M. Rao, and Elliot A. Stein. 2000. Practice-related functional activation changes in a working memory task. Microscopy research and technique 51, 1 (2000), 54-63.
16. Alan Gevins, Michael E. Smith, Harrison Leong, Linda McEvoy, Susan Whitfield, Robert Du, and Georgia Rush. 1998. Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Human Factors: The Journal of the Human Factors and Ergonomics Society 40, 1 (1998), 79-91.
17. Alan Gevins, Michael E. Smith, Linda McEvoy, and Daphne Yu. 1997. High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cerebral Cortex 7 (1997), 374-385. DOI: http://dx.doi.org/10.1093/cercor/7.4.374
18. Alexander Gundel and Glenn F. Wilson. 1992. Topographical changes in the ongoing EEG related to the difficulty of mental tasks. Brain topography 5, 1 (1992), 17-25.
19. Joshua Harrison, Kurtulus Izzetoglu, Hasan Ayaz, Ben Willems, Sehchang Hah, Ulf Ahlstrom, Hyun Woo, Patricia Shewokis, Scott C. Bunce, and Banu Onaral. 2014. Cognitive workload and learning assessment during the implementation of a next-generation air traffic control technology using functional near-infrared spectroscopy. IEEE Transactions on Human-Machine Systems 44, 4 (2014), 429-440.
20. Shamsi T Iqbal, Xianjun Sam Zheng, and Brian P Bailey. 2004. Task-evoked pupillary response to mental workload in human-computer interaction. Extended abstracts of the 2004 conference on Human factors and computing systems CHI 04 (2004), 1477. DOI: http://dx.doi.org/10.1145/985921.986094
21. Chen-Lin C. Kulik James A. Kulik and Robert L. Bangert-Drowns. 1985. Effectiveness of Computer-Based Education in Elementary schools. Computers in Human Behavior 1 (1985), 59-74.
22. Slava Kalyuga. 2003. The expertise reversal effect. Educational psychologist 38, 1 (2003), 23-31.
23. Slava Kalyuga. 2007. Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review 19, 4 (2007), 509-539.
24. Slava Kalyuga. 2011. Cognitive Load Theory: How Many Types of Load Does It Really Need? Educational Psychology Review 23 (2011), 1-19. DOI: http://dx.doi.org/10.1007/s10648-010--9150--7
25. Clare Kelly and Hugh Garavan. 2005. Human functional neuroimaging of brain changes associated with practice. Cerebral Cortex 15, August (2005), 1089-1102. DOI: http://dx.doi.org/10.1093/cercor/bhi005
26. Etienne Koechlin, Gianpaolo Basso, Pietro Pietrini, Seth Panzer, and Jordan Grafman. 1999. The role of the anterior prefrontal cortex in human cognition. Nature 399 (May 1999), 148-51. DOI: http://dx.doi.org/10.1038/20178
27. James A. Kulik. 1994. Technology Assessment in Education and Training. Eva Baker and Harold F. O'Neil Jr, Chapter Meta-analytic Studies of findings on computer-based instruction., 9-34.
28. Taija M M Lahtinen, Jukka P. Koskelo, Tomi Laitinen, and Tuomo K. Leino. 2007. Heart rate and performance during combat missions in a flight simulator. Aviation Space and Environmental Medicine 78, 4 (2007), 387-391.
29. Dara S. Manoach, Gottfried Schlaug, Bettina Siewert, David G. Darby, Benjamin M. Bly, Andrew Benfield, Robert R. Edelman, and Steven Warach. 1997. Prefrontal cortex fMRI signal changes are correlated with working memory load. Neuroreport 8, 2 (1997), 545-549. http://www.ncbi.nlm.nih.gov/pubmed/9080445
30. Richard E. Mayer. 2005. Cognitive theory of multimedia learning. (2005).
31. Roxana Moreno. 2010. Cognitive load theory: more food for thought. Instructional Science 38, November 2009 (2010), 135-141. DOI: http://dx.doi.org/10.1007/s11251-009--9122--9
32. Roger Nkambou, Riichiro Mizoguchi, and Jacqueline Bourdeau. 2010. Advances in intelligent tutoring systems. (2010).
33. Hyacinth S. Nwana. 1990. Intelligent tutoring systems: an overview. Artificial Intelligence Review 4, 4 (1990), 251-277. DOI: http://dx.doi.org/10.1007/BF00168958
34. Fred Paas, Juhani E. Tuovinen, Huib Tabbers, and Pascal W.M. Van Gerven. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist 38, 1 (2003), 63-71.
35. Fred G Paas. 1992. Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of educational psychology 84, 4 (1992), 429.
36. Evan M. Peck, Beste F. Yuksel, Alvitta Ottley, Rob J.K. Jacob, and Remco Chang. 2013. Using fNIRS Brain Sensing to Evaluate Information Visualization Interfaces. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (2013), 473-482.
37. Rosalind Picard. 1997. Affective computing 252 (1997).
38. Russell A Poldrack. 2000. Imaging brain plasticity: conceptual and methodological issues a theoretical review. Neuroimage 12, 1 (2000), 1-13.
39. Grega Repovs and Alan Baddeley. 2006. The multi-component model of working memory: Explorations in experimental cognitive psychology. Neuroscience 139, 1 (2006), 5-21.
40. Ron J.C.M. Salden, Fred Paas, Nick J. Broers, and Jeroen J.G. van Merrinboer. 2004. Mental effort and performance as determinants for the dynamic selection of learning tasks in air traffic control training. Instructional Science 32, 1--2 (2004), 153-172.
41. Wolfgang Schnotz and Christian Kurschner. 2007. A eeconsideration of cognitive load theory. Educational Psychology Review 19 (2007), 469-508. DOI: http://dx.doi.org/10.1007/s10648-007--9053--4
42. Holger Schultheis and Anthony Jameson. 2004. Assessing cognitive load in adaptive hypermedia systems: Physiological and behavioral methods. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 3137 (2004), 225-234. DOI: http://dx.doi.org/10.1007/b99480
43. Erin Treacy Solovey, Audrey Girouard, Krysta Chauncey, Leanne M Hirshfield, Angelo Sassaroli, Feng Zheng, Sergio Fantini, and Robert J K Jacob. 2009. Using fNIRS Brain Sensing in Realistic HCI Settings : Experiments and Guidelines. Proc. UIST 2009 (2009).
44. Erin Treacy Solovey, Paul Schermerhorn, Matthias Scheutz, Angelo Sassaroli, Sergio Fantini, and Robert J K Jacob. 2012. Brainput : Enhancing Interactive Systems with Streaming fNIRS Brain Input. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (2012).
45. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257-285.
46. John Sweller. 1999. Instructional Design in Technical Areas. ACER Press.
47. Daniel Szafir and Bilge Mutlu. 2012. Pay attention!: designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 11-20.
48. Daniel Szafir and Bilge Mutlu. 2013. ARTFul: adaptive review technology for flipped learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1001-1010.
49. Pascal W M Van Gerven, Fred Paas, Jeroen J G Van Merrienboer, and Henk G. Schmidt. 2004. Memory load and the cognitive pupillary response in aging. Psychophysiology 41 (2004), 167-174. DOI: http://dx.doi.org/10.1111/j.1469--8986.2003.00148.x
50. Arno Villringer and Britton Chance. 1997. Non-invasive optical spectroscopy and imaging of human brain function. Trends in Neurosciences 20 (1997). http://www.ncbi.nlm.nih.gov/pubmed/9347608
51. Lev Vygotsky. 1978. Interaction between learning and development. Mind and Society (1978), 79-91.
52. Christopher D. Wickens, Shaun. Hutchins, Thomas. Carolan, and John. Cumming. 2013. Effectiveness of Part-Task Training and Increasing-Difficulty Training Strategies: A Meta-Analysis Approach. Human Factors: The Journal of the Human Factors and Ergonomics Society (2013). DOI: http://dx.doi.org/10.1177/0018720812451994
53. Dennis C. Wightman and Gavan Lintern. 1985. Part-Task Training for Tracking and manual control. Human Factors 27, 3 (1985), 267-283.
54. Bin Xie and Gavriel Salvendy. 2000. Prediction of mental workload in single and multiple task environments. International journal of cognitive ergonomics 4 (2000), 213-242.
55. Beste F. Yuksel, Daniel Afergan, Evan M. Peck, Garth Griffin, Lane Harrison, Nick Chen, Remco Chang, and Rob J.K. Jacob. 2015. BRAAHMS: A Novel Adaptive Musical Interface Based on Users' Cognitive State. Proc. NIME 2015 (2015).

Index Terms

• Human-centered computing