A mobile phone based personal narrative system

ABSTRACT

Currently available commercial Augmentative and Alternative Communication (AAC) technology makes little use of computing power to improve the access to words and phrases for personal narrative, an essential part of social interaction. In this paper, we describe the development and evaluation of a mobile phone application to enable data collection for a personal narrative system for children with severe speech and physical impairments (SSPI). Based on user feedback from the previous project "How was School today?" we developed a modular system where school staff can use a mobile phone to track interaction with people and objects and user location at school. The phone also allows taking digital photographs and recording voice message sets by both school staff and parents/carers at home. These sets can be played back by the child for immediate narrative sharing similar to established AAC device interaction using sequential voice recorders. The mobile phone sends all the gathered data to a remote server. The data can then be used for automatic narrative generation on the child's PC based communication aid. Early results from the ongoing evaluation of the application in a special school with two participants and school staff show that staff were able to track interactions, record voice messages and take photographs. Location tracking was less successful, but was supplemented by timetable information. The participating children were able to play back voice messages and show photographs on the mobile phone for interactive narrative sharing using both direct and switch activated playback options.

References

1. Reiter, E., et al., Using NLG to Help Language-Impaired Users Tell Stories and Participate in Social Dialogues, in ENLG2009. 2009, Association for Computational Linguistics: Athens, Greece.
2. Black, R., et al., "How was School Today?" Evaluating the Personal-Narrative-Telling Prototype: Preliminary results, in Communication Matters Symposium 2009. 2009, Communication Matters: Leicester.
3. Reiter, E. An Architecture for Data-to-Text Systems. in ENLG-2007. 2007.
4. Vanderheiden, G.C., A journey through early augmentative communication and computer access. Journal of Rehabilitation Research and Development, 2002. 39(6): p. 39--53.
5. Waller, A., Communication Access to Conversational Narrative. Topics in Language Disorders, 2006. 26(3): p. 221--239.
6. McCabe, A. and C. Peterson, Getting the story: A longitudinal study of parental styles in eliciting narratives and developing narrative skill, in Developing narrative structure, A. McCabe and C. Peterson, Editors. 1991, Lawrence Erlbaum Associates: Hillsdale, NJ. p. 217--253.
7. Quasthoff, U.M. and K. Nikolaus, What makes a good story? Towards the production of conversational narratives, in Discourse Processing, A. Flammer and W. Kintsch, Editors. 1982, North-Holland Publishing Co.: Oxford.
8. Polkinghorne, D.E., Narrative configuration in qualitative analysis, in Life history and narrative, J.A. Hatch and R. Wisniewski, Editors. 1995, Routledge: London. p. 5--24.
9. Todman, J., et al., Whole utterance approaches in AAC. Augmentative & Alternative Communication, 2008. 24(3): p. 235--254.
10. Todman, J. and N.A. Alm, Modelling conversational pragmatics in communication aids. Journal of Pragmatics, 2003. 3: p. 523--538.
11. Dempster, M., N. Alm, and E. Reiter, Automatic generation of conversational utterances and narrative for Augmentative and Alternative Communication: a prototype system. , in Proceedings of NAACL-10 Workshop on Speech and Language Processing for Assistive Technology. 2010: Los Angeles, USA.
12. McCoy, K., C. Pennington, and A. Badman, Compansion: From research prototype to practical integration. Natural Language Engineering, 1998. 43(73--95).
13. Beukelman, D.R. and P. Mirenda, Augmentative and Alternative Communication: Management of Severe Communication Disorders in Children and Adults. 3rd ed. 2005, Baltimore: Paul H. Brookes Publishing Co.
14. Black, R., et al., Tell me about your day: creating novel access to personal narrative, in Communication Matters Symposium 2008. 2008, Communication Matters: Leicester.
15. Reddington, J. and N. Tintarev, Automatically generating stories from sensor data, in Proceedings of the 16th international conference on Intelligent user interfaces. 2011, ACM Palo Alto, CA, USA.
16. Musselwhite, C., K. Daswick, and S. Daswick. Self-Constructed Scripts. Tip of the Month 2005 {cited 2010 8 February}; Available from: http://www.aacintervention.com/selfconstructing%20scripts.pdf.
17. Black, R., et al., A New Tool to Support Interactive Narrative Experience for Children with Communication Disorders, in 14th Biennial Conference of the International Society for Augmentative and Alternative Communication. 2010, International Society for AAC: Barcelona, Spain.
18. Black, R., et al. Using NLG and Sensors to Support Personal Narrative for Children with Complex Communication Needs. in First Workshop on Speech and Language Processing for Assistive Technologies (SLPAT), Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2010. Los Angeles.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Natural language generation

• Human-centered computing

  • Human computer interaction (HCI)

  • Interaction design

    • Interaction design process and methods

      • User centered design

• Social and professional topics

  • Professional topics

    • Computing profession

      • Assistive technologies

  • User characteristics

    • People with disabilities

• Software and its engineering

  • Software notations and tools

    • Development frameworks and environments

      • Integrated and visual development environments