Automatic summarization of voicemail messages using lexical and prosodic features

ABSTRACT

This aticle presents trainable methods for extracting principal content words from voicemail messages. The short text summaries generated are suitable for mobile messaging applications. The system uses a set of classifiers to identify the summary words with each word described by a vector of lexical and prosodic features. We use an ROC-based algorithm, Parcel, to select input features (and classifiers). We have performed a series of objective and subjective evaluations using unseen data from two different speech recognition systems as well as human transcriptions of voicemail speech.

REFERENCES

• 1: Beckman, M. 1986. Stress and Non-Stress Accent. Foris Publications, Dordrecht, Holland/Riverton.
• 2: Chen, F. and Withgott, M. 1992. The use of emphasis to automatically summarize a spoken discourse. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'92). San Francisco, CA. 229--232.
• 3: Cordoba, R., Woodland, P. C., and Gales, M. J. F. 2002. Improving cross task performance using MMI training. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'02). Orlando, FL. 85--88.
• 4: H. P. Edmundson, New Methods in Automatic Extracting, Journal of the ACM (JACM), v.16 n.2, p.264-285, April 1969 [doi>10.1145/321510.321519]
• 5: Garofolo, J., Lard, J., and Voorhees, E. 2001. TREC-9 spoken document retrieval track: overview and results. In Proceedings of the 9th Text Retrieval Conference (TREC-9). Gaithersburg, MD.
• 6: Gotoh, Y. and Renals, S. 2000. Information extraction from broadcast news. Philosophical Trans. Royal Soc. London, Series A, 358, 1295--1310.
• 7: Hakkani-Tür, D., Tür, G., Stolcke, A., and Shriberg, E. 1999. Combining words and prosody for information extraction from speech. In Proceedings of Eurospeech. Budapest, Hungary. 1991--1994.
• 8: Hirschberg, J., Bacchiani, M., Hindle, D., Isenhour, P., Rosenberg, A., Stark, L., Stead, L., Whittaker, S., and Zamchick, G. 2001. SCANMail: Browsing and searching speech data by content. In Proceedings of Eurospeech. Aalborg, Denmark.
• 9: Hirschberg, J. and Nakatani, C. 1998. Acoustic indicators of topic segmentation. In Proceedings of International Conference on Spoken Language Processing (ICSLP'98). Sydney, Australia. 1255--1258.
• 10: Hori, C. and Furui, S. 2000. Improvements in automatic speech summarization and evaluation methods. In Proceedings of International Conference on Spoken Language Processing (ICSLP'00). Beijing, China. 326--329.
• 11: Jing Huang , Geoffrey Zweig , Mukund Padmanabhan, Information extraction from voicemail, Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, p.298-305, July 06-11, 2001, Toulouse, France [doi>10.3115/1073012.1073051]
• 12: Martin Jansche , Steven P. Abney, Information extraction from voicemail transcripts, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.320-327, July 06, 2002 [doi>10.3115/1118693.1118734]
• 13: Kato, Y. 1994. Voice message summary for voice services. In International Symposium on Speech, Image Processing and Neural Networks. Hong-Kong. 622--625.
• 14: Koumpis, K. 2002. Automatic voicemail summarisation for mobile messaging. Ph.D. thesis, University of Sheffield, UK.
• 15: Koumpis, K. 2004. Automatic categorization of voicemail transcripts using stochastic language models. In Proceedings of the 7th International Conference on Text, Speech and Language. Brno, Czech Republic. Lecture Notes in Computer Science.
• 16: Konstantinos Koumpis , Charalampos Ladas , Steve Renals, An Advanced Integrated Architecture for Wireless Voicemail Data Retrieval, Proceedings of the The 15th International Conference on Information Networking, p.403, January 31-February 02, 2001
• 17: Koumpis, K. and Renals, S. 2000. Transcription and summarization of voicemail speech. In Proceedings of International Conference on Spoken Language Processing (ICSLP'00). Beijing, China. 688--691.
• 18: Koumpis, K. and Renals, S. 2001. The role of prosody in a voicemail summarization system. In Proceedings of the ISCA Workshop on Prosody in Speech Recognition and Understanding. Red Bank, NJ. 87--92.
• 19: Koumpis, K., Renals, S., and Niranjan, M. 2001. Extractive summarization of voicemail using lexical and prosodic feature subset selection. In Proceedings of Eurospeech. Aalborg, Denmark. 2377--2380.
• 20: Kubala, F., Schwartz, R., Stone, R., and Weischedel, R. 1998. Named entity extraction from speech. In Proceedings of DARPA Broadcast News Transcription and Understanding Workshop. Lansdowne, VA.
• 21: Ladd, D. R. 1996. Intonational Phonology. Cambridge University Press, Cambridge, UK.
• 22: Maclay, H. and Osgood, C. 1959. Hesitation phenomena in spontaneous english speech. Word 1, 19--44.
• 23: Makhoul, J., Kubala, F., Schwartz, R., and Weischedel, R. 1999. Performance measures for information extraction. In Proceedings of the DARPA Broadcast News Workshop. Herndon, VA. 249--252.
• 24: Mani, I. 2001. Automatic Summarization. John Benjamins Publishing, Amsterdam, The Netherlands.
• 25: Mark T. Maybury , Inderjeet Mani, Advances in Automatic Text Summarization, MIT Press, Cambridge, MA, 1999
• 26: Medan, Y., Yair, E., and Chazan, D. 1991. Super resolution pitch determination of speech signal. IEEE Trans. Acoustics, Speech Signal Process. 39, 1, 40--48.
• 27: Morgan, N. and Bourlard, H. 1995. An introduction to hybrid HMM/connectionist continuous speech recognition. IEEE Signal Process. Mag., 25--42.
• 28: Morgan, N., Fosler, E., and Mirghafori, N. 1997. Speech recognition using on-line estimation of speaking rate. In Proceedings of Eurospeech. Rhodes, Greece. 2079--2082.
• 29: Padmanabhan, M., Eide, E., Ramabhardan, G., Ramaswany, G., and Bahl, L. 1998. Speech recognition performance on a voicemail transcription task. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'98). Seattle, WA. 913--916.
• 30: Paksoy, E., McCree, A., Viswananthan, V., and Linn, J. 1997. A variable-rate CELP coder for fast remote voicemail retrieval using a notebook computer. In Proceedings of the IEEE Workshop on Multimedia Signal Processing. Princeton, NJ. 119--124.
• 31: David D. Palmer , Mari Ostendorf , John D. Burger, Robust information extraction from automatically generated speech transcriptions, Speech Communication, v.32 n.1-2, p.95-109, Sept. 2000 [doi>10.1016/S0167-6393(00)00026-1 ]
• 32: Pierrehumbert, J. 1980. The phonology and phonetics of english intonation. Ph.D. thesis, MIT, Cambridge, MA.
• 33: Foster Provost , Tom Fawcett, Robust Classification for Imprecise Environments, Machine Learning, v.42 n.3, p.203-231, March 2001 [doi>10.1023/A:1007601015854 ]
• 34: Robertson, S. E. and Sparck Jones, K. 1997. Simple proven approaches to text retrieval. Tech. rep., TR-356, Cambridge University Computer Laboratory, Cambridge, UK.
• 35: A. J. Robinson , G. D. Cook , D. P. W. Ellis , E. Fosler-Lussier , S. J. Renals , D. A. G. Williams, Connectionist speech recognition of Broadcast News, Speech Communication, v.37 n.1-2, p.27-45, May 2002 [doi>10.1016/S0167-6393(01)00058-9 ]
• 36: Rohlicek, J. R., Ayuso, D., Bates, M., Bobrow, R., Boulanger, A., Gish, H., Jeanrenaud, P., Meteer, M., and Siu, M. 1992. Gisting conversational speech. In Proceedings of IEEE International Conference Acoustics, Speech, and Signal Processing (ICASSP'92). San Francisco, CA. 113--117.
• 37: Saon, G. and Padmanabhan, M. 2001. Data-driven approach to designing compound words for continuous speech recognition. IEEE Trans. Speech Audio Process. 9, 4, 327--332.
• 38: Scott, M., Niranjan, M., and Prager, R. 1998. Parcel: Feature subset selection in variable cost domains. Tech. rep., CUED TR-323, ftp://svr-ftp.eng.cam.ac.uk/pub/reports. Cambridge, UK.
• 39: Shriberg, E. 2001. To “errrr” is human: Ecology and acoustics of speech disfluencies. J. Int. Phonetic Ass. 31, 1, 153--169.
• 40: Elizabeth Shriberg , Andreas Stolcke , Dilek Hakkani-Tür , Gükhan Tür, Prosody-based automatic segmentation of speech into sentences and topics, Speech Communication, v.32 n.1-2, p.127-154, Sept. 2000 [doi>10.1016/S0167-6393(00)00028-5 ]
• 41: Mark Stevenson , Robert Gaizauskas, Using corpus-derived name lists for named entity recognition, Proceedings of the sixth conference on Applied natural language processing, p.290-295, April 29-May 04, 2000, Seattle, Washington [doi>10.3115/974147.974187]
• 42: Taylor, P., Caley, R., Black, A. W., and King, S. 1999. Edinburgh speech tools library. Tech. rep., ftp://ftp.cstr.ed.ac.uk. Edinburgh, UK.
• 43: Valenza, R., Robinson, T., Hickey, M., and Tucker, R. 1999. Summarization of spoken audio through information extraction. In Proceedings of ESCA Workshop on Accessing Information in Spoken Audio. Cambridge, UK. 111--116.
• 44: Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A. 1998. Evaluating spoken dialogue agents with PARADISE: Two case studies. Comput. Speech Lang. 12, 3, 317--347.
• 45: Warnke, V., Kompe, R., Niemann, H., and Nöth, E. 1997. Integrated dialog act segmentation and classification using prosodic features and language models. In Proceedings of Eurospeech. Rhodes, Greece. 207--210.
• 46: Williams, G. and Renals, S. 1999. Confidence measures from local posterior probability estimates. Comput. Speech Lang. 13, 395--411.
• 47: Klaus Zechner, Automatic generation of concise summaries of spoken dialogues in unrestricted domains, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.199-207, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383989]
• 48: Zweig, M. H. and Campbell, G. 1993. Receiver-operative characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chem. 39, 561--577.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors; Human information processing
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Audio input/output; Evaluation/methodology
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis; Text analysis

General Terms:
Languages

Keywords:
Voicemail, automatic summarization, feature subset selection, prosody, receiver operating characteristic, short message service