Multimodal interaction with speech and physical touch interface in a media center application

ABSTRACT

We present a multimodal media center interface based on a novel combination of new modalities. The application is based on a combination of a large high-definition display and a mobile phone. Users can interact with the system using speech input (speech recognition), physical touch (touching physical icons with the mobile phone), and gestures. We present the key results from a laboratory experiment where user expectations and actual usage experiences are compared.

References

1. Ailisto H, Pohjanheimo L, Välkkynen P, Strömmer E, Tuomisto T and Korhonen I (2006) Bridging the physical and virtual worlds by local connectivity-based physical selection, Personal Ubiquitous Computing, 10(6)333--344.
2. Arhippainen, L. and Tähti, M. (2003) Empirical Evaluation of User Experience in Two Adaptive Mobile Application Prototypes, in Proc. of MUM 2003, Norrköping, Sweden, pp. 27--34.
3. Battarbee, K. and Koskinen, I. Co-experience: user experience as interaction. CoDesign, 2005. 1(1), pp. 5--18.
4. Bederson, B. B. 2000. Fisheye menus. In Proceedings of the 13th Annual ACM Symposium on User interface Software and Technology (San Diego, California, United States, November 06--08, 2000). UIST '00. ACM, New York, NY, 217--225.
5. Bederson, B. B., Clamage, A., Czerwinski, M. P., and Robertson, G. G. 2004. DateLens: A fisheye calendar interface for PDAs. ACM Trans. Comput.-Hum. Interact. 11, 1 (Mar. 2004), 90--119.
6. Bederson, B. B., Grosjean, J., and Meyer, J. 2004. Toolkit Design for Interactive Structured Graphics. IEEE Trans. Softw. Eng. 30, 8 (Aug. 2004), 535--546.
7. Berglund, A., and Qvarfordt, P. 2003. Error Resolution Strategies for Interactive Television Speech Interfaces. In Proceedings of International Conference on Human-Computer Interaction (INTERACT '03). IFIP, Amsterdam, 2003, 105--112.
8. Broll G, Siorpaes S, Rukzio E, Paolucci M, Haamard J, Wagner M and Schmidt A (2007) Supporting Mobile Services Usage through Physical Mobile Interaction. Proc 5th IEEE Intl Conf on Pervasive Computing and Communications. White Plains, NY, USA, pp. 262--271.
9. Ferscha, A., Vogl, S., Emsenhuber, B., and Wally, B. 2007. Physical shortcuts for media remote controls. In Proceedings of the 2nd international Conference on intelligent Technologies For interactive Entertainment. ICST, Brussels, Belgium, 1--8.
10. Hassenzahl, M., Burmester, M, and Koller F. 2003. Attrak-Diff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J. Ziegler & G. Szwillus (Hrsg.), Mensch & Computer 2003. Interaktion in Bewegung. Stuttgart, Leipzig: B. G. Teubner, 187--196.
11. Hassenzahl, M. 2004. The thing and I: understanding the relationship between user and product. In Funology: From Usability To Enjoyment, M. A. Blythe, K. Overbeeke, A. F. Monk, and P. C. Wright, Eds. Kluwer Academic Publishers, Norwell, MA, 31--42.
12. Hassenzahl, M. The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 2004. 19(4), pp. 319--349.
13. Hassenzahl, M. and Tractinsky, N. (2006) User Experience -- A Research Agenda. Behaviour and Information Technology, Vol. 25, No. 2, pp. 91--97.
14. Ibrahim, A., and Johansson, P. 2003. Multimodal Dialogue Systems: A Case Study for Interactive TV. Carbonell, Noelle; Stephanidis, Constantine (Eds.) Universal Access. Theoretical Perspectives, Practice, and Experience, 7th ERCIM International Workshop on User Interfaces for All, Revised Papers. Springer, LNCS, Vol. 2615. 209--218.
15. Kankainen, A. (2003) UCPCD: User-Centered Product Concept Design, in Proc. of the 2003 conference on Designing for user experiences, ACM Press, pp. 1--13.
16. NFC Forum. Near Field Communication and the NFC Forum: The Keys to Truly Interoperable Communications. http://www.nfc-forum.org/resources/white_papers (Last Revised16.06.09).
17. NFC Forum. NFC Data Exchange Format (NDEF). Accessible from http://www.nfc-forum.org/specs/. Last access 16-06-09.
18. Riekki J., Sànchez I., and Pyykkönen M. Universal remote control for the smart world. In Proceedings of 5th International Conference on Ubiquitous Intelligence and Computing, UIC 2008, pages 563--577, Oslo, Norway, June 23--25 2008.
19. Sànchez I, Riekki J, Pyykkönen M (2008) Touch & Control: Interacting with Services by Touching RFID Tags. In Proceedings of the 2nd International Workshop on RFID Technology - Concepts, Applications, Challenges (IWRT 2008), In conjunction with ICEIS 2008. Barcelona, Spain, June 12--13, 2008. pp 53--62. http://www.ee.oulu.fi/research/isg/publications/ID/1272.
20. Schlömer, T., Poppinga, B., Henze, N., and Boll, S. 2008. Gesture recognition with a Wii controller. In Proceedings of the 2nd international Conference on Tangible and Embedded interaction (Bonn, Germany, February 18--20, 2008). TEI '08. ACM, New York, NY, 11--14.
21. Soronen H, Turunen M, Hakulinen J. 2008. Voice Commands in Home Environment - a Consumer Survey, In Proceedings of Interspeech 2008: 2078--2081.
22. Turunen, M., Melto, A., Hakulinen, J., Kainulainen, A., and Heimonen, T. User Expectations, User Experiences and Objective Metrics in a Multimodal Mobile Application. Proceedings of the Third Workshop on Speech in Mobile and Pervasive Environments, 2008.
23. Turunen, M., Hakulinen, J., Melto, A., Heimonen, T., Laivo, T., and Hella, J. SUXES -- User Experience Evaluation Method for Spoken and Multimodal Interaction. In Proceedings of Interspeech 2009.
24. Turunen, M., Hakulinen, J., Melto, A., Hella, J., Rajaniemi, J.-P., Mäkinen, E., Rantala, J., Heimonen, T., Laivo, T., Soronen, H., Hansen, M., Valkama, P. Miettinen, T., Raisamo, R. Speech-based and Multimodal Media Center for Different User Groups. In Proceedings of Interspeech 2009.
25. Wittenburg, K., Lanning, T., Schwenke, D., Shubin, H., and Vetro, A. 2006. The prospects for unrestricted speech input for TV content search. In Proceedings of the Working Conference on Advanced Visual interfaces (Venezia, Italy, May 23--26, 2006). AVI '06. ACM, New York, NY, 352--359.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interaction paradigms