ABSTRACT
The present paper reports on the design and performance of a novel dual-Wizard simulation infrastructure that has been used effectively to prototype next-generation adaptive and implicit multimodal interfaces for collaborative groupwork. This high-fidelity simulation infrastructure builds on past development of single-wizard simulation tools for multiparty multimodal interactions involving speech, pen, and visual input [1]. In the new infrastructure, a dual-wizard simulation environment was developed that supports (1) real-time tracking, analysis, and system adaptivity to a user's speech and pen paralinguistic signal features (e.g., speech amplitude, pen pressure), as well as the semantic content of their input. This simulation also supports (2) transparent user training to adapt their speech and pen signal features in a manner that enhances the reliability of system functioning, i.e., the design of mutually-adaptive interfaces. To accomplish these objectives, this new environment also is capable of handling (3) dynamic streaming digital pen input. We illustrate the performance of the simulation infrastructure during longitudinal empirical research in which a user-adaptive interface was designed for implicit system engagement based exclusively on users' speech amplitude and pen pressure [2]. While using this dual-wizard simulation method, the wizards responded successfully to over 3,000 user inputs with 95-98% accuracy and a joint wizard response time of less than 1.0 second during speech interactions and 1.65 seconds during pen interactions. Furthermore, the interactions they handled involved naturalistic multiparty meeting data in which high school students were engaged in peer tutoring, and all participants believed they were interacting with a fully functional system. This type of simulation capability enables a new level of flexibility and sophistication in multimodal interface design, including the development of implicit multimodal interfaces that place minimal cognitive load on users during mobile, educational, and other applications.
- Arthur, A., Lunsford, R., Wesson, M., and Oviatt, S. L. Prototyping novel collaborative multimodal systems: Simulation, data collection and analysis tools for the next decade, Proc. ICMI, 2006. Google ScholarDigital Library
- Oviatt, S. L., Swindells, C., and Arthur, A. Implicit user-adaptive system engagement in speech and pen interfaces, Conference on Human Factors in Computing Systems (CHI '08), CHI Letters, ACM: New York, N.Y., 2008, 969--978. Google ScholarDigital Library
- Cohen, P. R. and McGee, D. R. Tangible Multimodal Interfaces for Safety-Critical Applications. CACM 47(1), 2004, 41--46. Google ScholarDigital Library
- Dahlback, N., Jonsson, A., & Ahrenberg, L., Wizard-of-Oz Studies - Why and How, in Proc. of the Int'l Workshop on Intelligent User Interfaces, 1993. Google ScholarDigital Library
- Lunsford, R., and Oviatt, S. Human perception of intended addressee during computer-assisted meetings, Proc. of Int'l Conf. on Multimodal Interfaces, 2006, 20--27. Google ScholarDigital Library
- Martin, D., Cheyer, A. & Moran, D. The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence: An International Journal. 13(1-2), 1999.Google ScholarCross Ref
- Norrie, M. C., Signer, B. and Weibel, N., General Framework for the Rapid Development of Interactive Paper Applications, CoPADD 2006, Workshop on Collaborating over Paper and Digital Documents 2006Google Scholar
- Oviatt, S. L., Cohen, P. R., Fong, M. W., and Frank, M. P. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. In Ohala, J., et al., (Eds.), Proc. of the Int'l Conference on Spoken Language Processing, 2 Univ. of Alberta, 1992, 1351--1354.Google Scholar
- Oviatt, S. L., Coulston R., Tomko S., Xiao, B., Lunsford, R. Wesson, M. & Carmichael L., Toward a Theory of Organized Multimodal Integration Patterns during Human-Computer Interaction, Proc. of the Int'l Conf. on Multimodal Interfaces, ACM Press, 2003, 44--51. Google ScholarDigital Library
- Salber, D. & Coutaz, J., Applying the Wizard-of-Oz technique to the study of multimodal systems, Proc. of the European Workshop on HCI, 1993. Google ScholarDigital Library
- Yeh, R. B., Liao, C. Klemmer, S. Guimbretière F., Lee, B., Kakaradov, B., Stamberger, J., and Paepcke. A., ButterflyNet: A Mobile Capture and Access System for Field Biology Research. Proc. of CHI'06, pp. 571--580. Google ScholarDigital Library
Index Terms
- A high-performance dual-wizard infrastructure for designing speech, pen, and multimodal interfaces
Recommendations
Implicit user-adaptive system engagement in speech and pen interfaces
CHI '08: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsAs emphasis is placed on developing mobile, educational, and other applications that minimize cognitive load on users, it is becoming more essential to explore interfaces based on implicit engagement techniques so users can remain focused on their ...
Multimodal speech and pen interfaces
The Handbook of Multimodal-Multisensor InterfacesThis chapter describes interfaces that enable users to combine digital pen and speech input for interacting with computing systems. Such interfaces promise natural and efficient interaction, taking advantage of skills that users have developed over many ...
Wizard of oz for multimodal interfaces design: deployment considerations
HCI'07: Proceedings of the 12th international conference on Human-computer interaction: interaction design and usabilityThe use of Wizard of Oz (WOz) techniques for the acquisition of multimodal interaction patterns is common, but often relies on highly or fully simulated functionality. This paper suggests that a more operational WOz can benefit multimodal interaction ...
Comments