High-quality speech-to-speech translation for computer-aided language learning

ABSTRACT

This article describes our research on spoken language translation aimed toward the application of computer aids for second language acquisition. The translation framework is incorporated into a multilingual dialogue system in which a student is able to engage in natural spoken interaction with the system in the foreign language, while speaking a query in their native tongue at any time to obtain a spoken translation for language assistance. Thus the quality of the translation must be extremely high, but the domain is restricted. Experiments were conducted in the weather information domain with the scenario of a native English speaker learning Mandarin Chinese. We were able to utilize a large corpus of English weather-domain queries to explore and compare a variety of translation strategies: formal, example-based, and statistical. Translation quality was manually evaluated on a test set of 695 spontaneous utterances. The best speech translation performance (89.9% correct, 6.1% incorrect, and 4.0% rejected), is achieved by a system which combines the formal and example-based methods, using parsability by a domain-specific Chinese grammar as a rejection criterion.

REFERENCES

• 1: Hiyan Alshawi , Srinivas Bangalore , Shona Douglas, Head-Transducer Models for Speech Translation and TheirAutomatic Acquisition from Bilingual Data, Machine Translation, v.15 n.1-2, p.105-124, June 2000
• 2: Srinivas Bangalore , Giuseppe Riccardi, Stochastic Finite-State Models for Spoken Language MachineTranslation, Machine Translation, v.17 n.3, p.165-184, 2002 [doi>10.1023/B:COAT.0000010804.12581.96]
• 3: Baptist, L. and Seneff, S. 2000. Genesis-II: A versatile system for language generation in conversational system applications. In Proceedings of the International Conference on Spoken Language Processing. Beijing, China.
• 4: Peter F. Brown , John Cocke , Stephen A. Della Pietra , Vincent J. Della Pietra , Fredrick Jelinek , John D. Lafferty , Robert L. Mercer , Paul S. Roossin, A statistical approach to machine translation, Computational Linguistics, v.16 n.2, p.79-85, June 1990
• 5: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 6: Brown, R. D. 1999. Adding linguistic knowledge to a lexical example-based translation system. In Proceedings of the International Conference on Theoretical and Methodological Issues in Machine Translation. Chester, England.
• 7: Casacuberta, F., Ney, H., Och, F. J., Vidal, E., Vilar, J. M., Barrachina, S., Garcia-Varea, I., Llorens, D., Martinez, C., Molau, S., Nevado, F., Pastor, M., Pico, D., Sanchis, A., and Tillmann, C. 2004. Some approaches to statistical and finite-state speech-to-speech translation. Comput. Speech Lang. 18, 24--47.
• 8: Michael Collins, Three generative, lexicalised models for statistical parsing, Proceedings of the 35th annual meeting on Association for Computational Linguistics, p.16-23, July 07-12, 1997, Madrid, Spain
• 9: Cowan, B. 2004. PLUTO: A preprocessor for multilingual spoken language generation. M.S. thesis, MIT, Cambridge, MA.
• 10: Robert E. Frederking , Alan W. Black , Ralf D. Brown , Alexander Rudnicky , John Moody , Eric Steinbrecher, Speech translation on a tight budget without enough data, Proceedings of the ACL-02 workshop on Speech-to-speech translation: algorithms and systems, p.77-84, July 11, 2002 [doi>10.3115/1118656.1118667]
• 11: Gao, Y., Zhou, B., Diao, Z., Sorensen, J., Erdogan, H., and Sarikaya, R. 2002. A trainable approach for multilingual speech-to-speech translation system. In Proceedings of the Human Language Technology Conference. San Diego, CA.
• 12: Yuqing Gao , Bowen Zhou , Zijian Diao , Jeffrey Sorensen , Michael Picheny, MARS: A Statistical Semantic Parsing and Generation-Based Multilingual Automatic tRanslation System, Machine Translation, v.17 n.3, p.185-212, 2002 [doi>10.1023/B:COAT.0000010802.38267.29]
• 13: Glass, J. 2003. A probabilistic framework for segment-based speech recognition. Comput. Speech Lang. 17, 137--152.
• 14: Godfrey, J., Holliman, E., and McDaniel, J. 1992. Switchboard: Telephone speech corpus for research and development. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. San Francisco, CA, 517--520.
• 15: He, Y. and Young, S. 2003. A data-driven spoken language understanding system. In Proceedings of Automatic Speech Recognition and Understanding. St Thomas, US Virgin Islands.
• 16: Hitoshi Iida , Eiichiro Sumita , Osamu Furuse, Spoken-language translation method using examples, Proceedings of the 16th conference on Computational linguistics, August 05-09, 1996, Copenhagen, Denmark [doi>10.3115/993268.993369]
• 17: Koehn, P. 2004. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of the Association for Machine Translation in the Americas. Washington DC.
• 18: Philipp Koehn , Franz Josef Och , Daniel Marcu, Statistical phrase-based translation, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.48-54, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073462]
• 19: Shankar Kumar , William Byrne, A weighted finite state transducer implementation of the alignment template model for statistical machine translation, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.63-70, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073464]
• 20: Lee, J. and Seneff, S. 2004. Translingual grammar induction. In Proceedings of the International Conference on Spoken Language Processing. Jeju Island, Korea.
• 21: Lori Levin , Alon Lavie , Monika Woszczyna , Donna Gates , Marsal Gavaldá , Detlef Koll , Alex Waibel, The Janus-III Translation System: Speech-to-Speech Translation in Multiple Domains, Machine Translation, v.15 n.1-2, p.3-25, June 2000
• 22: Daniel Marcu, Towards a unified approach to memory- and statistical-based machine translation, Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, p.386-393, July 06-11, 2001, Toulouse, France [doi>10.3115/1073012.1073062]
• 23: Mitchell P. Marcus , Mary Ann Marcinkiewicz , Beatrice Santorini, Building a large annotated corpus of English: the penn treebank, Computational Linguistics, v.19 n.2, June 1993
• 24: Ney, H., Niessen, S., Och, F. J., Sawaf, H., Tillmann, C., and Vogel, S. 2000. Algorithms for statistical translation of spoken language. IEEE Trans. Speech Audio Proces. 8, 1, 24--36.
• 25: Franz Josef Och , Hermann Ney, A systematic comparison of various statistical alignment models, Computational Linguistics, v.29 n.1, p.19-51, March 2003 [doi>10.1162/089120103321337421]
• 26: Och, F. J., Tillmann, C., and Ney, H. 1999. Improved alignment models for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. College Park, MD, 20--28.
• 27: P. J. Price, Evaluation of spoken language systems: the ATIS domain, Proceedings of the workshop on Speech and Natural Language, p.91-95, June 24-27, 1990, Hidden Valley, Pennsylvania [doi>10.3115/116580.116612]
• 28: M. Rayner , D. Carter, Hybrid Language Processing in the Spoken Language Translator, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97) -Volume 1, p.107, April 21-24, 1997
• 29: Satoshi Sato, CTM: an example-based translation aid system, Proceedings of the 14th conference on Computational linguistics, August 23-28, 1992, Nantes, France [doi>10.3115/992424.992477]
• 30: Mark Seligman, Nine Issues in Speech Translation, Machine Translation, v.15 n.1-2, p.149-186, June 2000
• 31: Seneff, S. 1992a. Robust parsing for spoken language systems. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing. Vol. 1. San Francisco, CA, 189--192.
• 32: Stephanie Seneff, TINA: a natural language system for spoken language applications, Computational Linguistics, v.18 n.1, p.61-86, March 1992
• 33: Stephanie Seneff , Joseph Polifroni, Dialogue management in the Mercury flight reservation system, ANLP/NAACL 2000 Workshop on Conversational systems, p.11-16, May 04-04, 2000, Seattle, Washington [doi>10.3115/1117562.1117565]
• 34: Seneff, S., Wang, C., and Hazen, T. J. 2003. Automatic induction of n-gram language models from a natural language grammar. In Proceedings of the Eurospeech. Geneva, Switzerland.
• 35: Seneff, S., Wang, C., and Zhang, J. 2004. Spoken conversational interaction for language learning. In Proceedings of National Language Processing and Speech Technologies in Advanced Language Learning Systems. Venice, Italy.
• 36: Mitsuo Shimohata , Eiichiro Sumita , Yuji Matsumoto, Retrieving meaning-equivalent sentences for example-based rough translation, Proceedings of the HLT-NAACL 2003 Workshop on Building and using parallel texts: data driven machine translation and beyond, p.50-56, May 31-31, 2003, Edmonton, Canada [doi>10.3115/1118905.1118916]
• 37: Eiichiro Sumita, Example-based machine translation using DP-matching between word sequences, Proceedings of the workshop on Data-driven methods in machine translation, p.1-8, July 07-07, 2001, Toulouse, France [doi>10.3115/1118037.1118038]
• 38: Eiichiro Sumita , Hitoshi Iida, Experiments and prospects of Example-Based Machine Translation, Proceedings of the 29th annual meeting on Association for Computational Linguistics, p.185-192, June 18-21, 1991, Berkeley, California [doi>10.3115/981344.981368]
• 39: Min Tang , Xiaoqiang Luo , Salim Roukos, Active learning for statistical natural language parsing, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073105]
• 40: Veale, T. and Way, A. 1997. Gaijin: A template-driven bootstrapping approach to example-based machine translation. In Proceedings of the NMNLP. Sofia, Bulgaria.
• 41: Wang, C. and Seneff, S. 2004. High-quality speech translation for language learning. In Proceedings of Natural Language Processing: National Language Processing and Speech Technologies in Advanced Language Learning Systems. Venice, Italy.
• 42: Ye-Yi Wang , Alex Waibel, Modeling with structures in statistical machine translation, Proceedings of the 17th international conference on Computational linguistics, p.1357-1363, August 10-14, 1998, Montreal, Quebec, Canada
• 43: Richard Zens , Franz Josef Och , Hermann Ney, Phrase-Based Statistical Machine Translation, Proceedings of the 25th Annual German Conference on AI: Advances in Artificial Intelligence, p.18-32, September 16-20, 2002
• 44: Zue, V., Seneff, S., Glass, J., Polifroni, J., Pao, C., Hazen, T. J., and Hetherington, L. 2000. JUPITER: A telephone-based conversational interface for weather information. IEEE Trans. Speech Audio Process. 8, 1, 85--96.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language generation; Machine translation

  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

General Terms:
Algorithms, Experimentation, Languages

Keywords:
Speech translation, computer-aided language learning, dialogue systems, machine translation