ABSTRACT
This paper demonstrates a multimodal system called Sketch-Thru-Plan (STP) that enables users to speak and draw doctrinal language and symbols in order to create courses of action. We argue that STP can meet many of the challenges inherent in building user interfaces for operations planning and command-and-control. The system is being transitioned to military organizations for use in planning courses of action.
- R.A. Bolt. "Voice and Gesture at the Graphics Interface." ACM Comp. Graphics, 14(3), 1980, pp. 262--270. Google ScholarDigital Library
- A. Cheyer, and L. Julia, Multimodal maps: An agent-based approach, International Conf. on Cooperative Multimodal Communication, Springer, 1995, pp. 24--26.Google Scholar
- P.R. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, J. Clow. QuickSet: Multimodal interaction for distributed applications Proc. of the 5th ACM Int'l Conf. on Multimedia, 31--40. Google ScholarDigital Library
- M. Johnston, P. R Cohen, D. McGee, S. L. Oviatt, J. A Pittman, I. Smith, Unification-based multimodal integration. Proc.of the Eighth Conference of the Assoc. for Computational Linguistics, 1997, 281--288. Google ScholarDigital Library
- D. B. Moran, A. J. Cheyer, L. E. Julia, D. L. Martin, S. Park, Multimodal user interfaces in the Open Agent Architecture, Proc. of the 2nd Int'l Conference on Intelligent User Intefaces, ACM Press, 1997, pp. 61--68. Google ScholarDigital Library
- S. L. Oviatt , Multimodal interactive maps: Designing for human performance. Human-Computer Interaction 12 (1), 1997, pp. 93--129. Google ScholarDigital Library
- Oviatt, S. L., and Cohen, P. R. Perceptual user interfaces: multimodal interfaces that process what comes naturally, CACM 43 (3), 45--53, 2000 Google ScholarDigital Library
- R. Sharma, M. Yeasin, N. Krahnstoever, I. Rauschert, G. Cai, I. Brewer, A. M. MacEachren, K. Sengupta. Speech--Gesture Driven Multimodal Interfaces for Crisis Management., Proc. IEEE, 91(9), 2003, 1327--1354.Google ScholarCross Ref
- M.W., Salisbury, J.H. Hendrickson, T.L Lammers, C. Fu, S.A. Talk and Draw: Bundling speech and graphics, Moody, IEEE Computer, 23(8), August, 1990, pp. 59--65 Google ScholarDigital Library
Index Terms
Demonstration of sketch-thru-plan: a multimodal interface for command and control
Recommendations
Steps towards collaborative multimodal dialogue (sustained contribution award)
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionThis talk will discuss progress in building collaborative multimodal systems, both systems that offer a collaborative interface that augments human performance, and autonomous systems with which one can collaborate. To begin, I discuss what we will ...
Speech and sketching for multimodal design
IUI '04: Proceedings of the 9th international conference on Intelligent user interfacesWhile sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. ...
Beyond attention: the role of deictic gesture in intention recognition in multimodal conversational interfaces
IUI '08: Proceedings of the 13th international conference on Intelligent user interfacesIn a multimodal conversational interface supporting speech and deictic gesture, deictic gestures on the graphical display have been traditionally used to identify user attention, for example, through reference resolution. Since the context of the ...
Comments