The effect of task conditions on the comprehensibility of synthetic speech

ABSTRACT

A study was conducted with 78 subjects to evaluate the comprehensibility of synthetic speech for various tasks ranging from short, simple e-mail messages to longer news articles on mostly obscure topics. Comprehension accuracy for each subject was measured for synthetic speech and for recorded human speech. Half the subjects were allowed to take notes while listening, the other half were not. Findings show that there was no significant difference in comprehension of synthetic speech among the five different text-to-speech engines used. Those subjects that did not take notes performed significantly worse for all synthetic voice tasks when compared to recorded speech tasks. Performance for synthetic speech in the non note-taking condition degraded as the task got longer and more complex. When taking notes, subjects also did significantly worse within the synthetic voice condition averaged across all six tasks. However, average performance scores for the last three tasks in this condition show comparable results for human and synthetic speech, reflective of a training effect.

REFERENCES

• 1: Francis, A.L., Nusbaum, H.C. (1999). Evaluating the quality of Synthetic Speech. In Gardner-Bonneau, D. (Ed.), Human Factors and Voice Interactive Systems (pp. 63 - 97)
• 2: Greenspan, S. L., Nusbaum, H.C., and Pisoni D.B. (1988). Perception of synthetic speech produced by rule: lntellibility of eight text-to-speech systems. Behavioral Research Methods, Instruments, and Computers, 18 100-107
• 3: Klatt, D. H. (1987). Review of text-to-speech conversion for English. The Journal of the Acoustical Society of America. September 1987 (pp. 737- 793)
• 4: Pisoni, D.B. and Hunnieutt, S. (1980). Perceptual Evaluation of MITalk: The MIT unrestricted text-to-speech system. IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 572-575) New York
• 5: Pisoni D.B., Nusbaum H.C. & Greene B.G. (1985). Perception of synthetic speech generated by rule. Proceedings of the 1EEE 73:1665-1676
• 6: James V. Ralston , David B. Pisoni , Scott E. Lively , Beth G. Greene , John W. Mullennix, Comprehension of synthetic speech produced by rule: word monitoring and sentence-by-sentence listening times, Human Factors, v.33 n.4, p.471-491, Aug. 1991
• 7: Ralston, J.V., Pisoni, D.B., and Muliermix, J.W. (1995). Perception and comprehension of speech. In Syrdal, A.K., Bennett, R.W., Greenspan, S.L. (Eds.), Applied Speech Technology (pp. 233-288) Boca Raton: CRC Press
• 8: Van Bezooijen, R. & van Heuven, V. (1998) Assessment of Synthesis System. In Gibbon, D., Moore, R. and Winski,R. (Eds.) , Volume 111." Spoken Language System Assessment (pp. 1671481} - 249 {563}) Mouton de Gruyer
• 9: www.toefl.org
• 10: www.genmagic.com/portico/portico__home.shtml
• 11: www.webley.com

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

General Terms:
Design, Experimentation, Human Factors, Languages, Management, Measurement, Performance, Theory

Keywords:
comprehension, synthetic speech, text-to-speech, user study