picoTrans: An intelligent icon-driven interface for cross-lingual communication

Abstract

picoTrans is a prototype system that introduces a novel icon-based paradigm for cross-lingual communication on mobile devices. Our approach marries a machine translation system with the popular picture book. Users interact with picoTrans by pointing at pictures as if it were a picture book; the system generates natural language from these icons and the user is able to interact with the icon sequence to refine the meaning of the words that are generated. When users are satisfied that the sentence generated represents what they wish to express, they tap a translate button and picoTrans displays the translation. Structuring the process of communication in this way has many advantages. First, tapping icons is a very natural method of user input on mobile devices; typing is cumbersome and speech input errorful. Second, the sequence of icons which is annotated both with pictures and bilingually with words is meaningful to both users, and it opens up a second channel of communication between them that conveys the gist of what is being expressed. We performed a number of evaluations of picoTrans to determine: its coverage of a corpus of in-domain sentences; the input efficiency in terms of the number of key presses required relative to text entry; and users' overall impressions of using the system compared to using a picture book. Our results show that we are able to cover 74% of the expressions in our test corpus using a 2000-icon set; we believe that this icon set size is realistic for a mobile device. We also found that picoTrans requires fewer key presses than typing the input and that the system is able to predict the correct, intended natural language sentence from the icon sequence most of the time, making user interaction with the icon sequence often unnecessary. In the user evaluation, we found that in general users prefer using picoTrans and are able to communicate more rapidly and expressively. Furthermore, users had more confidence that they were able to communicate effectively using picoTrans.

References

1. Barnard, K., Johnson, M., and Forsyth, D. 2003. Word sense disambiguation with pictures. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL) Workshop on Learning Word Meaning from Non-Linguistic Data Series (HLT-NAACL-LWM'04). Vol. 6. Association for Computational Linguistics, 1--5.
2. Baur, D., Boring, S., and Butz, A. 2010. Rush: Repeated recommendations on mobile devices. In Proceedings of the 14^th International Conference on Intelligent User Interfaces (IUI'10). ACM Press, New York, 91--100.
3. Doddington, G. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Human Language Technologies Conference (HLT).
4. Finch, A., Dixon, P., and Sumita, E. 2011a. Integrating models derived from non-parametric bayesian co-segmentation into a statistical machine transliteration system. In Proceedings of the Named Entities Workshop. Asian Federation of Natural Language Processing, 23--27.
5. Finch, A. and Sumita, E. 2008. Phrase-Based machine transliteration. In Proceedings of the 3^rd International Joint Conference on Natural Language Processing (IJCNLP). Vol. 1.
6. Finch, A. and Sumita, E. 2010. A bayesian model of bilingual segmentation for transliteration. In Proceedings of the 7^th International Workshop on Spoken Language Translation (IWSLT), M. Federico, I. Lane, M. Paul, and F. Yvon, Eds., 259--266.
7. Finch, A. M., Song, W., Tanaka-Ishii, K., and Sumita, E. 2011b. Picotrans: Using pictures as input for machine translation on mobile devices. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 2614--2619.
8. Finch, A. M., Song, W., Tanaka-Ishii, K., and Sumita, E. 2011c. Source language generation from pictures for machine translation on mobile devices. In Proceedings of the 8^th International Workshop on Natural Language Processing and Cognitive Science (NLPCS).
9. Flanagan, C. 2008. Me No Speak: China, 2nd ed. Me No Speak.
10. Graf, D. 2009. Point it: Traveller's Language Kit - The Original Picture Dictionary - Bigger and Better (English, Spanish, French, Italian, German and Russian Edition). Graf Editions.
11. Hu, C., Bederson, B. B., and Resnik, P. 2010. Translation by iterative collaboration between monolingual users. In Proceedings of the Conference on Graphics Interface (GI'10). Canadian Information Processing Society, 39--46.
12. Kikui, G., Sumita, E., Takezawa, T., and Yamamoto, S. 2003. Creating corpora for speech-to-speech translation. In Proceedings of the 8^th European Conference on Speech Communication and Technology (EUROSPEECH'03). 381--384.
13. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowa, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45^th Annual Meeting of the Association for Computational Linguistics Demo and Poster Sessions (ACL'07). 177--180.
14. Kudo, T. 2008. MeCab. http://mecab.sourceforge.net/.
15. Ma, X. and Cook, P. R. 2009. How well do visual verbs work in daily communication for young and old adults? In Proceedings of the 27^th International Conference on Human factors in Computing Systems (CHI'09). ACM Press, New York, 361--364.
16. Mackenzie, S. and Tanaka-Ishii, K., Eds. 2007. Text Entry Systems—Accessibility, Mobility, Universality. Morgan Kaufmann, San Fransisco, CA.
17. Meader, J. 1995. The Wordless Travel Book: Point at These Pictures to Communicate with Anyone. Ten Speed Press.
18. Mihalcea, R. and Leong, C. W. 2008. Toward communicating simple sentences using pictorial representations. Mach. Translation 22, 153--173.
19. Murphy, J. and Cameron, L. 2008. The effectiveness of talking mats with people with intellectual disability. Brit. J. Learn. Disabil. 36, 4, 232--241.
20. Noguchi, Y. and Hayashi, K. 2003. Yubisashi Travel Conversation 48: Australian (旅の指さし会話帳48オーストリア: オーストリア(ドイツ)語). Information Center Publishing (情報センター出版局).
21. Och, F. J. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41^st Meeting of the Association for Computational Linguistics (ACL'03).
22. Och, F. J. and Ney, H. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of the 40^th Annual Meeting of the Association for Computational Linguistics (ACL'02). 295--302.
23. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. 2001. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40^th Annual Meeting on Association for Computational Linguistics (ACL'02). Association for Computational Linguistics, 311--318.
24. Paul, M., Okuma, H., Yamamoto, H., Sumita, E., Matsuda, S., Shimizu, T., and Nakamura, S. 2008. Multilingual mobile-phone translation services for world travelers. In Proceedings of the 22^nd International Conference on Computational Linguistics, Companion Volume. 165--168.
25. Paul, M., Yamamoto, H., Sumita, E., and Nakamura, S. 2009. On the importance of pivot language selection for statistical machine translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL) Short Papers. 221--224.
26. picTrans. 2010. A simple picture-based translation system. http://www.7zillion.com/iPhone/PicTrans/.
27. Sirisena, A. 2002. Honours report, Christchurch. Tech. rep., Department of Computer Science, University of Canterbury.
28. Song, W., Finch, A. M., Tanaka-Ishii, K., and Sumita, E. 2011. Picotrans: An icon-driven user interface for machine translation on mobile devices. In Proceedings of the 16^th International Conference on Intelligent User Interfaces (IUI'11). ACM Press, New York, 23--32.
29. Stillman. 2007. Kwikpoint International Translator (English, Spanish, French, Italian, German, Japanese, Russian, Ukrainian, Chinese, Hindi, Tamil, Telug, Kannada, Malayalam, Gujarati, Bengali and Korean Edition). Kwikpoint.
30. Stolcke, A. 2002. SRILM - An extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing. Vol. 2. 901--904.
31. TexTra. 2010. Text translator by NICT. http://mastar.jp/translation/textra-en.html.
32. Vogel, D. and Baudisch, P. 2007. Shift: A technique for operating pen-based interfaces using touch. In Proceedings of the Conference on Human Factors in Computing Systems (CHI'07). ACM Press, New York, 657--666.
33. VoiceTra. 2010. Voice translator by NICT. http://mastar.jp/translation/voicetra-en.html.
34. Warrink, G. 2007. ICOON Global Picture Dictionary (English, Spanish, French, Italian, German, Japanese, Russian, Chinese and Hindi Edition). Amber Press.
35. Yubisashi. 2013. Yubisashi. Information center publishing. http://www.yubisashi.com/free/t/iphone/.
36. Zhu, X., Goldberg, A. B., Eldawy, M., Dyer, C. R., and Strock, B. 2007. A text-to-picture synthesis system for augmenting communication. In Proceedings of the 22^nd International Conference on Artificial Intelligence 2. 1590--1595.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Machine translation