Blending Human and Artificial Intelligence to Support Autistic Children’s Social Communication Skills

Abstract

This article examines the educational efficacy of a learning environment in which children diagnosed with Autism Spectrum Conditions (ASC) engage in social interactions with an artificially intelligent (AI) virtual agent and where a human practitioner acts in support of the interactions. A multi-site intervention study in schools across the UK was conducted with 29 children with ASC and learning difficulties, aged 4--14 years old. For reasons related to data completeness and amount of exposure to the AI environment, data for 15 children was included in the analysis. The analysis revealed a significant increase in the proportion of social responses made by ASC children to human practitioners. The number of initiations made to human practitioners and to the virtual agent by the ASC children also increased numerically over the course of the sessions. However, due to large individual differences within the ASC group, this did not reach significance. Although no evidence of transfer to the real-world post-test was shown, anecdotal evidence of classroom transfer was reported. The work presented in this article offers an important contribution to the growing body of research in the context of AI technology design and use for autism intervention in real school contexts. Specifically, the work highlights key methodological challenges and opportunities in this area by leveraging interdisciplinary insights in a way that (i) bridges between educational interventions and intelligent technology design practices, (ii) considers the design of technology as well as the design of its use (context and procedures) on par with one another, and (iii) includes design contributions from different stakeholders, including children with and without ASC diagnosis, educational practitioners, and researchers.

References

1. B. Abirached, Y. Zhang, J. Aggarwal, B. Tamersoy, T. Fernandes, J. Miranda, and V. Orvalho. 2009. Improving communication skills of children with ASDs through interaction with virtual characters. In Proceedings of the IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH). 1--4.
2. A. Alcorn, H. Pain, G. Rajendran, T. Smith, O. Lemon, K. Porayska-Pomsta, M. E. Foster, K. Avramides, C. Frauenberger, and S. Bernardini. 2011. Social communication between virtual characters and children with autism. In Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED’11). Springer-Verlag, Berlin, 7--14.
3. A. M. Alcorn. 2016. Embedding novel and surprising elements in touch-screen games for children with autism: creating experiences ‘worth communicating about’. Ph.D. Dissertation. University of Edinburgh, Edinburgh, Scotland.
4. K. Anderson, E. André, T. Baur, S. Bernardini, M. Chollet, E. Chryssafidou, I. Damian, C. Ennis, A. Egges, P. Gebhard, H. Jones, M. Ochs, C. Pelachaud, K. Porayska-Pomsta, P. Rizzo, and N. Sabouret. 2013. The TARDIS framework: Intelligent virtual agents for social coaching in job interviews. In Advances in Computer Entertainment. Springer, 476--491.
5. A. Anwar, M. Rahman, S. Ferdous, S. Anik, and S. Ahmed. 2011. AA computer game based approach for increasing fluency in the speech of the autistic children. In Proceedings of the 11th IEEE International Conference on Advanced Learning Technologies (ICALT’11). 17--18.
6. APA. 2013. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5). American Psychiatric Association.
7. R. Aylett, N. Vannini, E. Andre, A. Paiva, S. Enz, and L. Hall. 2009. But it was another country: Agents and intercultural empathy. In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (I/International Foundation for Autonomous Agents and Multiagent Systems.). Vol. 1, 329--336.
8. R. S. Baker. 2016. Stupid tutoring systems, intelligent humans.International Journal of Artificial Intelligence in Education 26, 2 (2016), 600--614.
9. E. Barakova, G. van Wanrooij, R. van Limpt, and M. Menting. 2007. Using an emergent system concept in designing interactive games for autistic children. In Proceedings of the 6th International Conference on Interaction Design and Children. (IDC’07). ACM, 73--76.
10. L. Barnard-Brak, A. Brewer, S. Chesnut, D. Richman, and A. M. Schaeffer. 2016. The sensitivity and specificity of the social communication questionnaire for autism spectrum with respect to age. Autism Research 9, 8 (2016), 838--845.
11. L. Bartoli, C. Corradi, F. Garzotto, and M. Valoriani. 2013. Exploring motion-based touchless games for autistic children?s learning. In Proceedings of the ACM Interaction Design and Children Conference (IDC’13). ACM, New York.
12. A. Battocchi, F. Pianesi, D. Tomasini, M. Zancanaro, G. Esposito, P. Venuti, A. Ben Sasson, E. Gal, and P. L. Weiss. 2013. Collaborative puzzle game: A tabletop interactive game for fostering collaboration in children with autism spectrum disorders (ASD). In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS’09). ACM, 197--204.
13. N. Bauminger. 2002. The facilitation of social-emotional understanding and social interaction in high-functioning children with autism: Intervention outcomes. Journal of Autism and Developmental Disorders 32 (2002), 283--298.
14. R. Beaumont and K. Sofronoff. 2008. A multi-component social skills intervention for children with asperger syndrome: The junior detective training program.Journal of Child Psychology and Psychiatry 49 (2008), 743--753.
15. S. Bernardini and K. Porayska-Pomsta. 2013. Planning-based social partners for children with autism. In Proceedings of the Twenty Third International Conference on Automated Planning and Scheduling (ICAPS-13). AAAI Press, Rome, Itay.
16. S. Bernardini, K. Porayska-Pomsta, and Smith T. J. 2014. Echoes: An intelligent serious game for fostering social communication in children with autism. Information Sciences 264 (2014), 41--60.
17. G. Biesta. 2007. Why “what works” won’t work: Evidence-based practice and the democratic deficit in educational research.Educational Theory 57, 1 (2007), 1--22.
18. G. Biesta. 2013. The Beautiful Risk of Education. Boulder and London: Paradigm Publishers.
19. A. Bosseler and D. Massaro. 2003. Development and evaluation of a computer- animated tutor for vocabulary and language learning in children with autism. IJournal of Autism and Developmental Disorders 33, 6 (2003), 653--672.
20. M. O. Cavazza, F. Charles, and S. J. Mead. 2002. Character-based interactive storytelling. IEEE Intelligent Systems 17, 4 (2002), 17--24.
21. H.-M. Chiang and M. Carter. 2008. Spontaneity of communication in individuals with autism. Journal of Autism and Developmental Disorders 38, 4 (2008), 693--705.
22. S. Cobb. 2007. Virtual environments supporting learning and communication in special needs education. Topics in Language Disorders 27, 3 (2007), 211--225.
23. Mihaly Csikszentmihalyi. 1991. Flow: The Psychology of Optimal Experience. Harper Perennial, New York, NY. Retrieved from http://www.amazon.com/gp/product/0060920432/ref=si3_rdr_bb_product/104-4616565-4570345
24. K. Dautenhahn and I. Werry. 2004. Towards interactive robots in autism therapy: Background, motivation and challenges. Pragmatics and Cognition 12, 1 (2004), 1--35.
25. M. Dennis, D. J. Francis, P. T. Cirino, R. Schachar, M. A. Barnes, and J. M. Fletcher. 2009. Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders. Journal of the International Neuropsychological Society 15, 3 (2009), 331--343.
26. J. Dewey. 1998. Experience and Nature. Dover Publications Inc.
27. J. Dias and A. Paiva. 2005. Feeling and reasoning: A computational model for emotional characters. In Progress in Artificial Intelligence. Lecture Notes in Computer Science, vol. 3808. Springer Berlin, Heidelberg, 127--140.
28. H. E. Dingfelder and D. S. Mandell. 2011. Bridging the research-to-practice gap: An application of diffusion of innovation theory. Journal of Autism and Developmental Disorders 41, 5 (2011), 597--609.
29. W. Farr, N. Yuill, and H. Raffle. 2010. Social benefits of a tangible user interface for children with autistic spectrum conditions. Autism 14, 3 (2010), 237--252.
30. S. L. Finkelstein, A. Nickel, L. Harrison, E. A. Suma, and T. Barnes. 2009. cMotion: A new game design to teach emotion recognition and programming logic to children using virtual humans. In Proceedings of the 2009 IEEE Virtual Reality Conference. 249--250.
31. S. Fletcher-Watson. 2014. A targeted review of computer-assisted learning for people with autism spectrum disorder: Towards a consistent methodology. Review Journal of Autism and Developmental Disorders 1, 2 (2014), 87--100.
32. S. Fletcher-Watson, H. Pain, S. Hammond, A. Humphry, and H. McConachie. 2016. Designing for young children with autism spectrum disorder: A case study of an iPad app. International Journal of Child-Computer Interaction 7 (2016), 1--14.
33. M. E. Foster, K. Avramides, S. Bernardini, J. Chen, C. Frauenberger, O. Lemon, and K. Porayska-Pomsta. 2010. Supporting children?s social communication skills through interactive narratives with virtual characters. In Proceedings of ACM Multimedia. ACM, Florence, Italy.
34. C. Frauenberger, J. Good, A. Alcorn, and H. Pain. 2013. Conversing through and about technologies: Design critique as an opportunity to engage children with autism and broaden research(er) perspectives. International Journal of Child-Computer Interaction 1, 2 (2013), 38--49.
35. C. Frauenberger, J. Good, and W. Keay-Bright. 2010. Phenomenology, a framework for participatory design. In Proceedings of the 11th Biennial Participatory Design Conference (PDC’10). ACM, New York, NY, 187--190.
36. C. Frauenberger, J. Good, and W. Keay-Bright. 2011. Designing technology for children with special needs: Bridging perspectives through participatory design. CoDesign 7, 1 (2011), 1--28.
37. O. Golan, E. Ashwin, Y. Granader, S. McClintock, K. Day, V. Leggett, and S. Baron-Cohen. 2010. A multi-component social skills intervention for children with asperger syndrome: The junior detective training program. Journal of Autism and Developmental Disorders 40 (2010), 269--279.
38. M. Goodwin. 2008. Enhancing and accelerating the pace of autism research and treatment: The promise of developing innovative technology. Focus on Autism and Other Developmental Disabilities 23, 2 (2008).
39. C. Gray and J. Garand. 1993. Social stories: Improving responses of students with autism with accurate social information. Focus on Autism and Other Developmental Disabilities 8, 1 (1993), 1--10.
40. O. Grynszpan, J.-C. Martin, and J. Nadel. 2008. Multimedia interfaces for users with high functioning autism: An empirical investigation. International Journal on Human-Computer Studies 66, 8 (2008), 628--639.
41. O. Grynszpan, P. L. T. Weiss, F. Perez-Diaz, and E. Gal. 2014. Innovative technology-based interventions for autism spectrum disorders: A meta-analysis. Autism 18, 4 (2014), 851--871.
42. K. Guldberg. 2017. Evidence based practice in autism educational research: Can we bridge the research and practice gap? Oxford Review of Education 43, 4 (2017).
43. K. Guldberg, S. Parsons, K. Porayska-Pomsta, and W. Keay-Bright. 2017. Challenging the knowledge transfer orthodoxy: Knowledge co-construction in technology enhanced learning for children with autism. British Educational Research Journal 43, 2 (2017), 384--413.
44. M. Hauck, D. Fein, L. Waterhouse, and C. Feinstein. 1995. Social initiations by autistic children to adults and other children. Journal of Autism and Developmental Disorders 25 (1995), 579--595.
45. G. Herrera, F. Alcantud, R. Jordan, A. Blanquer, G. Labajo, and C. De Pablo. 2008. Development of symbolic play through the use of virtual reality tools for children with autism spectrum disorders. Two case studies. Sage Publications and The National Autistic Society 12, 2 (2008), 143--157.
46. G. Herrera, Casas X., J. Sevilla, L. Rosa, C. Pardo, J. Plaza, R. Jordan, and S. Le Groux. 2012. Pictogram room: Natural interaction technologies to aid in the development of children with autism. Annuary of Clinical and Health Psychology 8 (2012), 39--44.
47. S. Holt and N. Yuill. 2017. Tablets for two: How dual tablets can facilitate other-awareness and communication in learning disabled children with autism. International Journal of Child-Computer Interaction 11 (2017), 72--82.
48. I. M. Hopkins, M. W. Gower, T. A. Perez, D. S. Smith, F. R. Amthor, and et al. 2011. Avatar assistant: Improving social skills in students with an ASD through a computer-based intervention. Journal of Autism and Developmental Disorders 41, 11 (2011), 1543--1555.
49. J. P. Hourcade, S. R. Williams, E. A. Miller, K. E. Huebner, and L. J. Liang. 2013. Evaluation of tablet apps to encourage social interaction in children with autism spectrum disorders. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ACM, New York, 3197--3206.
50. N. Kandalaft, M. R. Didehbani, D. C. Krawczyk, T. T. Allen, and S. B. Chapman. 2013. Virtual reality social cognition training for young adults with high-functioning autism, journal of autism and developmental disorders. Journal of Autism and Developmental Disorders 43, 1 (2013), 34--44.
51. P. Kenny, H. Arno, J. Gratch, W. Swartout, D. R. Traum, S. Marsella, and D. Piepol. 2007. Building interactive virtual humans for training environments. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC).
52. H. Kozima, M. Michalowski, and C. Nakagawa. 2009. Keepon: A playful robot for research, therapy, and entertainment. International Journal of Social Robotics 1, 1 (2009), 3--18.
53. C. Lord, M. Rutter, P. DiLavore, S. Risi, K. Gotham, and S. Bishop. 2012. Autism Diagnostic Observation Schedule: ADOS-2. Western Psychological Services Torrance.
54. L. Malinverni, J. Mora, V. Padillo, A. Hervas, and N. Pares. 2014. Pico?s adventure: A kinect game to promote social initiation in children with autism spectrum disorder. In Proceedings of the ITASD International Conference (IDC’16). ACM, Paris, France.
55. D. Massaro. 2006. Embodied agents in language learning for children with language challenges. In Proceedings of the 10th International Conference on Computers Helping People with Special Needs. (ICCHP?06). Springer-Verlag, Berlin, Heidelberg, 809--816.
56. J. Mora-Guiard, C. Crowell, N. Pares, and Heaton. P. 2016. Lands of fog: Helping children with autism in social interaction through full-body interactive experience. In Proceedings of the ACM Interaction Design and Children Conference (IDC’16). ACM, Manchester, UK.
57. L. Mottron. 2004. Matching strategies in cognitive research with individuals with high-functioning autism: Current practices, instrument biases, and recommendations. Journal of Autism and Developmental Disorders 34, 1 (2004), 19--27.
58. P. Mundy, C. Delgado, J. Block, M. Venezia, A. Hogan, and J. Seibert. 2003. Early Social Communication Scales. University of Miami, Coral Gables, FL.
59. NAS 2016. What is autism? Retrieved November 12, 2016 from http://www.autism.org.uk/about/what-is.aspx.
60. A. Ortony, G L. Clore, and A. Collins. 1988. The Cognitive Structure of Emotions. Cambridge University Press.
61. S. Parsons, T. Charman, R. Faulkner, J. Ragan, S. Wallace, and K. Wittemeyer. 2013. Commentary? bridging the research and practice gap in autism: The importance of creating research partnerships with schools. Autism 17, 3 (2013), 268--280.
62. S. Parsons and S. Cobb. 2011. State-of-the-art of virtual reality technologies for children on the autism spectrum. European Journal of Special Needs Education 26, 3 (2011), 355--366.
63. S. Parsons, K. Guldberg, A. MacLeod, G. Jones, A. Prunty, and T. Balfe. 2007. International review of the evidence on best practice in educational provision for children on the autism spectrum. European Journal of Special Needs Education 26, 1 (2007), 47--63.
64. S. Parsons, K. Guldberg, K. Porayska-Pomsta, and R. Lee. 2015. Digital stories as a method for evidence-based practice and knowledge co-creation in technology-enhanced learning for children with autism. International Journal of Research and Method in Education 38, 3 (2015), 247--271.
65. S. Parsons and P. Mitchell. 2017. The potential of virtual reality in social skills training for people with autistic spectrum disorders. Journal of Intellectual Disability Research 46, 5 (2017), 430--443.
66. R. C. Pennington. 2010. Computer-assisted instruction for teaching academic skills to students with autism spectrum disorders: A review of literature. Focus on Autism and Other Developmental Disabilities 25, 1 (2010), 239--248.
67. C. Pole and M. Morrison. 2003. Ethnography for Education. Open University Press.
68. K. Porayska-Pomsta. 2016. AI as a methodology for supporting educational praxis and teachers? Metacognition. International Journal of Artificial Intelligence in Education 26, 2 (2016), 679--700.
69. K. Porayska-Pomsta, K. Anderson, S. Bernardini, K. Guldberg, T. Smith, L. Kossivaki, S. Hodgings, and I. Lowe. 2013. Building and an intelligent, authorable serious game for autistic children and their carers. In Advances in Computer Entertainment. Springer, Cham, 456--475.
70. K. Porayska-Pomsta, C. Frauenberger, H. Pain, G. Rajendran, T. Smith, R. Menzies, M. E. Foster, A. Alcorn, S. Wass, S. Bernardini, K. Avramides, W. Keay-Bright, J. Chen, A. Waller, K. Guldberg, J. Good, and O. Lemon. 2011. Developing technology for autism: An interdisciplinary approach. Personal Ubiquitous Compututing 16, 2 (Feb. 2011), 117--127.
71. B. M. Prizant, A. M. Wetherby, E. Rubin, and A. C. Laurent. 2003. The SCERTS model: A transactional, family-centered approach to enhancing communication and socioemotional ability in children with autism spectrum disorder. Infants and Young Children 16, 4 (2003), 296--316.
72. B.M. Prizant, A.M. Wetherby, E. Rubin, A.C. Laurent, and P.J. Rydell. 2006. The SCERTS® Model: A Comprehensive Educational Approach for Children with Autism Spectrum Disorders. Brookes.
73. M. M. Rahman, S. Ferdous, S. I. Ahmed, and A. Anwar. 2011. Speech development of autistic children by interactive computer games. Interactive Technology and Smart Education 8, 4 (2011), 208--223.
74. G. Rajendran. 2013. Virtual environments and autism: A developmental psychopathological approach. Journal of Computer Assisted Learning 29, 4 (2013), 334--347.
75. S. Ramdoss, R. Lang, A. Mulloy, J. Franco, M. O’Reilly, R. Didden, and G. Lancioni. 2011a. Use of computer-based interventions to teach communication skills to children with autism spectrum disorders: A systematic review. Journal of Behavioral Education 20, 1 (2011), 55--76.
76. S. Ramdoss, A. Mulloy, R. Lang, M. O’Reilly, J. Sigafoos, and et al. Lancioni, G.2011b. Use of computer-based interventions to improve literacy skills in students with autism spectrum disorders: A systematic review. Research in Autism Spectrum Disorders 5, 4 (2011), 1306--1318.
77. P. A. Rao, D. C. Beidel, and M. J. Murray. 2008. Character-based interactive storytelling. Journal of Autism and Developmental Disorders 38, 2 (Feb. 2008), 353--361.
78. V. S. Rao, V. Raman, and A. V. Mysore. 2015. Issues related to obtaining intelligence quotient-matched controls in autism research. Indian Journal of Psychological Medicine 37, 2 (2015), 149.
79. B. Reichow, F. R. Volkmar, and D. V. Chiccetti. 2008. Development of the evaluative method for evaluating and determining evidence-based practices in autism. Journal of Autism and Developmental Disorders 38 (2008), 1311--1319.
80. G. Reynhout and M. Carter. 2009. The use of social stories by teachers and their perceived efficacy. Research in Autism Spectrum Disorders 3, 1 (2009), 232--251.
81. S. J. Russell and P. Norvig. 2003. Artificial Intelligence: A Modern Approach (2nd ed.) Prentice Hall.
82. N. J. Salkind. 2010. Encyclopedia of Research Design. Sage Research Methods.
83. D. Scattone, D. H. Tingstrom, and S. M. Wilczynski. 2006. Increasing appropriate social interactions of children with autism spectrum disorders using social StoriesTM. Focus on Autism and Other Developmental Disabilities 21, 4 (2006), 211--222.
84. B. Schuller, E. Marchi, S. Baron-Cohen, H. OReilly, P. Robinson, I. Davies, O. Golan, S. Friedenson, S. Tal, S. Newman, N. Meir, R. Shillo, A. Camurri, S. Piana, S. Blte, D. Lundqvist, B. Berggren, S. A., and N. Sullings. 2013. IASC-inclusion: Interactive emotion games for social inclusion of children with autism spectrum conditions. In Proceedings 1st International Workshop on Intelligent Digital Games for Empowerment and Inclusion (IDGEI’13).
85. C. A. Smith and R. S. Lazarus. 1990. Emotion and adaptation. In Handbook of Personality: Theory and Research. I. Vlahavas, D. Vrakas, and L.A. Pervin (Eds.). Guilford, New York, Chapter 23, 609--637.
86. A. C. Stahmer, L. Schreimban, and A. B. Cunningham. 2011. Toward a technology of treatment individualization for young children with autism spectrum disorders. Brain Research 1380 (2011), 229--239.
87. D. C. Strickland, D. McAllister, C. D. Coles, and S. Osborne. 2007. An evolution of virtual reality training designs for children with autism and fetal alcohol spectrum disorders. Topics in Language Disorders 27, 3 (2007), 226--241.
88. J. Suh, A. Orinstein, M. Barton, C. M. Chen, I. M. Eigsti, N. Ramirez-Esparza, and D. Fein. 2016. Ratings of broader autism phenotype and personality traits in optimal outcomes from autism. Journal of Autism and Developmental Disorders 46, 11 (2016), 3505--3518.
89. A. Tartaro and J. Cassell. 2008. Playing with virtual peers: Bootstrapping contingent discourse in children with autism. In Proceedings of the International Conference on the Learning Sciences (ICLA’08). ACM, 382--389.
90. Report US National Research Council. 2002. Scientific Research in Education. Technical Report. US National Research Council.
91. S. Wass and K. Porayska-Pomsta. 2014. The uses of cognitive training technologies in the treatment of autism spectrum disorders. Autism: International Journal of Research and Practice. 18, 8 (2014), 851--871.
92. C. Whalen and L. Schreibman. 2003. Joint attention training for children with autism using behavior modification procedures. Journal of Child Psychology and Psychiatry 44, 3 (2003), 456--468.
93. M. Wooldridge and N. R. Jennings. 1995. Intelligent agents: Theory and practice. Knowledge Engineering Review 10, 2 (1995), 115--152.
94. B. Woolf. 2008. Building Intelligent Tutoring Systems. Morgan Kaufman.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • Field studies

• Social and professional topics

  • User characteristics

    • Age

      • Children