Helping students keep up with real-time captions by pausing and highlighting

ABSTRACT

We explore methods for improving the readability of real- time captions by allowing users to more easily switch their gaze between multiple visual information sources. Real-time captioning provides deaf and hard of hearing (DHH) users with access to spoken content during live events, and the web has allowed these services to be provided via remotely- located captioning services, and for web content itself. However, despite caption benefits, spoken language reading rates often result in DHH users falling behind spoken content, especially when the audio is paired with visual references. This is particularly true in classroom settings, where multi-modal content is the norm, and captions are often poorly positioned in the room, relative to speakers. Additionally, this accommodation can benefit other students who face temporary or "situational" disabilities such as listening to unfamiliar speech accents, or if a student is in a location with poor acoustics.

In this paper, we explore pausing and highlighting as a means of helping DHH students keep up with live classroom content by helping them track their place when reading text involving visual references. Our experiments show that by providing users with a tool to more easily track their place in a transcript while viewing live video, it is possible for them to follow visual content that might otherwise have been missed. Both pausing and highlighting have a positive impact on students' scores on comprehension tests, but highlighting is preferred to pausing, and yields nearly twice as large of an improvement. We then discuss several issues with captioning that we observed during our design process and user study, and then suggest future work that builds on these insights.

References

1. M. S. Bernstein, J. R. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of UIST 2011, pages 33--42.
2. J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh. Vizwiz: nearly real-time answers to visual questions. In Proceedings of UIST 2010, pages 333--342. 2010.
3. A. C. Cavender, J. P. Bigham, and R. E. Ladner. ClassInFocus. In Proceedings of ASSETS 2009, pages 67--74.
4. M. Federico and M. Furini. Enhancing learning accessibility through fully automatic captioning. In Proceedings of W4A 2012, page 1.
5. C. Jensema. Viewer reaction to different television captioning speeds. American annals of the deaf, 143(4):318--24, Oct. 1998.
6. R. Kheir and T. Way. Inclusion of deaf students in computer science classes using real-time speech transcription. In Proceedings of ITiCSE 2007, pages 261--265.
7. R. S. Kushalnagar, P. Kushalnagar, and G. Manganelli. Collaborative Gaze Cues for Deaf Students. In Proceedings of DuET Workshop at CSCW 2012.
8. R. S. Kushalnagar, W. S. Lasecki, and J. P. Bigham. Accessibility Evaluation of Classroom Captions. TACCESS, 5(3):1--25, 2013.
9. R. S. Kushalnagar, B. P. Trager, and K. B. Beiter. Accessible Viewing Devices for Deaf and Hard of Hearing Students. In Convention of American Instructors of the Deaf. 2013.
10. W. S. Lasecki and J. P. Bigham. Online quality control for real-time crowd captioning. In Proceedings of ASSETS 2012.
11. W. S. Lasecki and J. P. Bigham. Interactive Crowds: Real-Time Crowdsourcing and Crowd Agents. Chapter In Handbook of Human Computation. Ed. P. Michelucci. Springer, 2013.
12. W. S. Lasecki, K. I. Murray, S. White, R. C. Miller, and J. P. Bigham. Real-time crowd control of existing interfaces. In Proceedings UIST 2011, pages 23--32.
13. W. S. Lasecki, C. D. Miller, and J. P. Bigham. Warping time for more effective real-time crowdsourcing. In Proceedings of CHI 2013, pages 2033--2036.
14. W. S. Lasecki, C. D. Miller, A. Sadilek, A. Abumoussa, D. Borrello, R. Kushalnagar, and J. P. Bigham. Real-time captioning by groups of non-experts. In In Proceedings of UIST 2012. pages 23--34.
15. M. Marschark, J. B. Pelz, C. Convertino, P. Sapere, M. E. Arndt, and R. Seewagen. Classroom Interpreting and Visual Information Processing in Mainstream Education for Deaf Students: Live or Memorex(R)? In American Educational Research Journal, 42(4):727--761, Jan. 2005.
16. I. Naim, D. Gildea, W. Lasecki, and J. P. Bigham. Text alignment for real-time crowd captioning. In Proceedings of NAACL-HLT 2013, pages 201--210.
17. M. D. Tyler, C. Jones, L. Grebennikov, G. Leigh, W. Noble, and D. Burnham. Effect of caption rate on the comprehension of educational television programmes by deaf school students. Deafness & Education International, 11(3):152--162, 2009.
18. M. Wald. Crowdsourcing correction of speech recognition captioning errors. In Proceedings of W4A 2011, page 1.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing profession

      • Assistive technologies

  • User characteristics

    • People with disabilities