Teaching Social Communication Skills Through Human-Agent Interaction

Abstract

There are a large number of computer-based systems that aim to train and improve social skills. However, most of these do not resemble the training regimens used by human instructors. In this article, we propose a computer-based training system that follows the procedure of social skills training (SST), a well-established method to decrease human anxiety and discomfort in social interaction, and acquire social skills. We attempt to automate the process of SST by developing a dialogue system named the automated social skills trainer, which teaches social communication skills through human-agent interaction. The system includes a virtual avatar that recognizes user speech and language information and gives feedback to users. Its design is based on conventional SST performed by human participants, including defining target skills, modeling, role-play, feedback, reinforcement, and homework. We performed a series of three experiments investigating (1) the advantages of using computer-based training systems compared to human-human interaction (HHI) by subjectively evaluating nervousness, ease of talking, and ability to talk well; (2) the relationship between speech language features and human social skills; and (3) the effect of computer-based training using our proposed system. Results of our first experiment show that interaction with an avatar decreases nervousness and increases the user's subjective impression of his or her ability to talk well compared to interaction with an unfamiliar person. The experimental evaluation measuring the relationship between social skill and speech and language features shows that these features have a relationship with social skills. Finally, experiments measuring the effect of performing SST with the proposed application show that participants significantly improve their skill, as assessed by separate evaluators, by using the system for 50 minutes. A user survey also shows that the users thought our system is useful and easy to use, and that interaction with the avatar felt similar to HHI.

References

1. Yaser Adi, Amanda Kiloran, Kulsum Janmohamed, and Sarah Stewart-Brown. 2007. Systematic Review of the Effectiveness of Interventions to Promote Mental Wellbeing in Primary Schools. Report 1: Universal approaches: Non-Violence Related Outcomes. Coventry, University of Warwick.
2. American Psychiatric Association. 2013. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). Author.
3. Simon Baron-Cohen, Jennifer Richler, Dheraj Bisarya, Nhishanth Gurunathan, and Sally Wheelwright. 2003. The systemizing quotient: An investigation of adults with Asperger syndrome or high-functining autism, and normal sex differences. Philosophical Transactions of the Royal Society of London B: Biological Sciences 358, 1430, 361--374.
4. Simon Baron-Cohen, Sally Wheelwright, Richard Skinner, Joanne Martin, and Emma Clubley. 2001. The autism-spectrum quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders 31, 1, 5--17.
5. Nirit Bauminger. 2002. The facilitation of social-emotional understanding and social interaction in high-functioning children with autism: Intervention outcomes. Journal of Autism and Developmental Disorders 32, 4, 283--298.
6. Amy L. Baylor and Yanghee Kim. 2004. Pedagogical agent design: The impact of agent realism, gender, ethnicity, and instructional role. In Intelligent Tutoring Systems. Springer, Berlin, Germany, 592--603.
7. Alan S. Bellack. 2004. Social Skills Training for Schizophrenia: A Step-by-Step Guide. Guilford Press, New York, NY.
8. J. Bishop. 2003. The Internet for educating individuals with social impairments. Journal of Computer Assisted Learning 19, 4, 546--556.
9. Daniel Bone, Matthew P. Black, Chi-Chun Lee, Marian E. Williams, Pat Levitt, Sungbok Lee, and Shrikanth Narayanan. 2012. Spontaneous-speech acoustic-prosodic features of children with autism and the interacting psychologist. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (INTERSPEECH'12).
10. Yoram S. Bonneh, Yoram Levanon, Omrit Dean-Pardo, Lan Lossos, and Yael Adini. 2010. Abnormal speech spectrum and increased pitch variability in young autistic children. Frontiers in Human Neuroscience 4, 237.
11. Jacob Cohen, Patricia Cohen, Stephen G. West, and Leona S. Aiken. 2013. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Routledge.
12. M. Davis, K. Dautenhahn, C. L. Nehaniv, and S. D. Powell. 2004. Towards an interactive system facilitating therapeutic narrative elicitation in autism. In Proceedings of the 3rd International Conference on Narrative and Interactive Learning Environments.
13. Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American Statistical Association 56, 293, 52--64.
14. Nancy Eisenberg, William Damon, and Richard M. Lerner. 2006. Social, Emotional, and Personality Development. John Wiley & Sons.
15. Cecilia A. Essau, Beatriz Olaya, Satoko Sasagawa, Jayshree Pithia, Diane Bray, and Thomas H. Ollendick. 2014. Integrating video-feedback and cognitive preparation, social skills training and behavioural activation in a cognitive behavioural therapy in the treatment of childhood anxiety. Journal of Affective Disorders 167, 261--267.
16. M. H. Hanson. 1995. Glottal Characteristics of Female Speakers. Ph.D. Dissertation. Harvard University, Cambridge, MA.
17. Peter A. Heeman, Rebecca Lunsford, Ethan Selfridge, Lois Black, and Jan Van Santen. 2010. Autism and interactional aspects of dialogue. In Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 249--252.
18. Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W. Picard. 2013. Mach: My automated conversation coach. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, NY, 697--706.
19. Mohammed Ehsan Hoque and Rosalind W. Picard. 2014. Rich nonverbal sensing technology for automated social skills training. Computer 47, 4, 28--35.
20. Leo Kanner. 1943. Autistic disturbances of affective contact. Nervous Child 2, 217--250.
21. Géza Kiss and Jan P. H. van Santen. 2013. Estimating speaker-specific intonation patterns using the linear alignment model. In Proceedings of the 14th Annual Conference of the International Speech Communication Association (INTERSPEECH'13). 354--358.
22. Géza Kiss, Jan P. H. van Santen, Emily Tucker Prud'hommeaux, and Lois M. Black. 2012. Quantitative analysis of pitch in speech of children with neurodevelopmental disorders. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (INTERSPEECH'12).
23. Jina Lee, Zhiyang Wang, and Stacy Marsella. 2010. Evaluating models of speaker head nods for virtual agents. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1. 1257--1264.
24. R. P. Liberman and C. J. Wallace. 1990. Social and Independent Living Skills: Basic Conversation Skills Module. Robert Paul Liberman, Camarillo, CA.
25. Robert Paul Liberman. 1987. Social and Independent Living Skills. UCLA Psychiatric Rehabilitation Consultants, Northridge, CA.
26. Joanne McCann and Sue Peppé. 2003. Prosody in autism spectrum disorders: A critical review. International Journal of Language and Communication Disorders 38, 4, 325--350.
27. David Moore, Paul McGrath, and John Thorpe. 2000. Computer-aided learning for people with autism—a framework for research and development. Innovations in Education and Teaching International 37, 3, 218--228.
28. Iftekhar Naim, M. Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015. Automated prediction and analysis of job interview performance: The role of what you say and how you say it. In Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG'15), Vol. 1. IEEE, Los Alamitos, CA, 1--6.
29. Sarah Parsons and Peter Mitchell. 2002. The potential of virtual reality in social skills training for people with autistic spectrum disorders. Journal of Intellectual Disability Research 46, 5, 430--443.
30. James W. Pennebaker, Roger J. Booth, and Martha E. Francis. 2007. Linguistic Inquiry and Word Count: LIWC {Computer Software}. Available at http://liwc.wpengine.com.
31. Toni Rietveld and Roeland Van Hout. 1993. Statistical Techniques for the Study of Language and Language Behaviour. Walter de Gruyter.
32. Masoud Rouhizadeh, Emily Prud'hommeaux, Brian Roark, and Jan Van Santen. 2013. Distributional semantic models for the evaluation of disordered language. In Proceedings of the Conference of the Association for Computational Linguistics. 709.
33. Jan P. H. Santen, Richard W. Sproat, and Alison Presmanes Hill. 2013. Quantifying repetitive speech in autism spectrum disorders and language impairment. Autism Research 6, 5, 372--383.
34. Björn Schuller, Erik Marchi, Simon Baron-Cohen, Helen O'Reilly, Delia Pigat, Peter Robinson, and Ian Daves. 2014. The state of play of ASC-inclusion: An integrated Internet-based environment for social inclusion of children with autism spectrum conditions. arXiv:1403.5912.
35. Miriam Silver and Peter Oakes. 2001. Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5, 3, 299--316.
36. Petr Slovák and Geraldine Fitzpatrick. 2015. Teaching and developing social and emotional skills with technology. ACM Transactions on Computer-Human Interaction 22, 4, 19.
37. Petr Slovák, Ran Gilad-Bachrach, and Geraldine Fitzpatrick. 2015. Designing social and emotional skills training: The challenges and opportunities for technology support. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, 2797--2800.
38. Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, and Satoshi Nakamura. 2014. Linguistic and acoustic features for automatic identification of autism spectrum disorders in children's narrative. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. 88.
39. Hiroki Tanaka, Sakriani Sakti, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2015. Automated social skills trainer. In Proceedings of the 20th International Conference on Intelligent User Interfaces. ACM, New York, NY, 17--27.
40. Amy Vaughan Van Hecke, Jocelyn Lebow, Elgiz Bal, Damon Lamb, Emily Harden, Alexis Kramer, John Denver, Olga Bazhenova, and Stephen W. Porges. 2009. Electroencephalogram and heart rate regulation to familiar and unfamiliar people in children with autism spectrum disorders. Child Development 80, 4, 1118--1133.
41. Akio Wakabayashi, Simon Baron-Cohen, Sally Wheelwright, and Yoshikuni Tojo. 2006. The autism-spectrum quotient (AQ) in Japan: A cross-cultural comparison. Journal of Autism and Developmental Disorders 36, 2, 263--270.
42. Charles J. Wallace, Connie J. Nelson, Robert Paul Liberman, Robert A. Aitchison, David Lukoff, John P. Elder, and Chris Ferris. 1980. A review and critique of social skills training with schizophrenic patients. Schizophrenia Bulletin 6, 1, 42.
43. Donna Williams. 1992. Nobody Nowhere. Doubleday.

Index Terms

• Applied computing

  • Education

• Human-centered computing

  • Human computer interaction (HCI)