A transformational approach for multimodal web user interfaces based on UsiXML

ABSTRACT

A transformational approach for developing multimodal web user interfaces is presented that progressively moves from a task model and a domain model to a final user interface. This approach consists of three steps: deriving one or many abstract user interfaces from a task model and a domain model, deriving one or many concrete user interfaces from each abstract one, and producing the code of the corresponding final user interfaces. To ensure these steps, transformations are encoded as graph transformations performed on the involved models expressed in their graph equivalent. For each step, a graph grammar gathers relevant graph transformations for accomplishing the sub-steps. The final user interface is multimodal as it involves graphical (keyboard, mouse) and vocal interaction. The approach outlined in the paper is illustrated throughout a running example for a graphical interface, a vocal interface, and two multimodal interfaces with graphical and vocal predominances, respectively.

REFERENCES

• 1: Aït-Ameur, Y., Breholáe, B., Girard, P., Guittet, L., Jambon, F.: Formal Verification and Validation of Interactive Systems Specifications: From Informal Specifications to Formal Validation. In: R. Jacquart (ed.), Proc. of 18th IFIP World Computer Congress. Kluwer Academics Publishers, Dordrecht (2004) 61--76.
• 2: Berti, S., Paternò, F.: Model-based Design of Speech Interfaces. In: Proc. of 10th Int. Conf. on Design, Specification, and Verification of Interactive Systems DSV-IS'2003 (Madeira, 4-6 June 2003). LNCS, Vol. 2844. Springer Verlag, Berlin (2003) 231--244.
• 3: Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., and Vanderdonckt, J.: June 2003, A Unifying Reference Framework for Multi-Target User Interfaces. Interacting with Computers 15(3) 289--308.
• 4: Chen, M., Luo, J., Dong, S.: Task-Oriented Synergistic Multimodality. In: Proc. of 1st Int. Conf. on Multimodal Interface ICMI'96 (Beijing, October 1996).
• 5: Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.: Four Easy Pieces for Assessing the Usability of Multimodal Interaction: the CARE properties. In: Proc. of 5th IFIP TC 13 Int. Conf. on Human-Computer Interaction Interact'95, 115--120.
• 6: Göbel, S., Buchholz, S., Ziegert, T., Schill, A.: Device Independent Representation of Web-based Dialogs and Contents. In: Proc. of the IEEE Youth Forum in Computer Science and Engineering Yuforic'01 (Valencia, November 2001). IEEE Computer Press.
• 7: Hastie, H., Johnston, M., Ehlen, P.: Context-sensitive Help for Multimodal Dialogue. In: Proc. of the 6th ACM Int. Conf. on Multimodal Interfaces ICMI'2004. ACM Press, New York (2004) 93--98.
• 8: IBM Voice Toolkit for WebSphere Studio. Int. Business Machines (10 September 2004) http://www-306.ibm.com/software/ pervasive/voice_toolkit/.
• 9: IBM Multimodal Browser. Int. Business Machines (10 September 2004) http://www-306.ibm.com/software/pervasive/mul timodal/.
• 10: Kouichi Katsurada , Yusaku Nakamura , Hirobumi Yamada , Tsuneo Nitta, XISL: a language for describing multimodal interaction scenarios, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958483]
• 11: Larson, J.A., Raman, T.V., Raggett, D.: Multimodal Interaction Framework, W3C Note. W3 Consortium (6 May 2003), http://www. w3.org/TR/mmi-framework.
• 12: Quentin Limbourg , Jean Vanderdonckt, Addressing the mapping problem in user interface design with UsiXML, Proceedings of the 3rd annual conference on Task models and diagrams, November 15-16, 2004, Prague, Czech Republic [doi>10.1145/1045446.1045474]
• 13: Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., Lopez-Jaquero, V.: UsiXML: a Language Supporting Multi-Path Development of User Interfaces. In: Proc. of EHCI-DSVIS'2004 (Hamburg, 11-13 July 2004). Kluwer Academics, Dordrecht (2005) 207--228.
• 14: Limbourg, Q.: Multi-path Development of User Interfaces. Ph.D. thesis. Universitá catholique de Louvain, Louvain-la-Neuve (2004).
• 15: Maes, S.: Position Statement for Multimodal Workshop. In: Proc. of W3C Workshop on Multimodal interaction MMI'2004 (Sophia Antipolis, 19-20 July 2004). W3C (2004) http://www.w3.org/2004/02/ mmi-workshop/maes-oracle.pdf.
• 16: Montero, F., López-Jaquero, V., Vanderdonckt, J., Gonzalez, P., Lozano, M.D., Solving the Mapping Problem in User Interface Design by Seamless Integration in IdealXML, Proc. of DSV-IS'2005, Springer-Verlag, Berlin, 2005, to appear.
• 17: Wolfgang Mueller , Robbie Schaefer , Steffen Bleul, Interactive Multimodal User Interfaces for Mobile Devices, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 9, p.90286.1, January 05-08, 2004
• 18: Palanque, Ph., Schyn, A., A Model-Based Approach for Engineering Multimodal Interactive. In: Proc. of 9th IFIP TC13 Int. Conf. on Human-Computer Interaction Interact'2003 (Zurich, 1-5 September 2003). IOS Press, Amsterdam (2003) 543--550.
• 19: Rainer Simon , Michael Jank Kapsch , Florian Wegscheider, A generic uiml vocabulary for device- and modality independent user interfaces, Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, May 19-21, 2004, New York, NY, USA [doi>10.1145/1013367.1013512]
• 20: X+V-Authoring, Deploying and Consuming Multimodal Services. Versatile Multimodal Solutions (2004) http://www.sys-con.com/ xml/article.cfm?id=615.
• 21: Ziegert, T., Lauff, M., Heuser, L.: Device Independent Web Applications - The Author Once - Display Everywhere Approach. Proc. of 4th Int. Conf. on Web Engineering ICWE'04 (Munich, 28-30 July 2004). Lecture Notes in Computer Science, Vol. 3140. Springer-Verlag, Berlin (2004) 244--255.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: Computer-aided software engineering (CASE)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces; Evolutionary prototyping

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping; Graphical user interfaces (GUI); Voice I/O; Interaction styles (e.g., commands, menus, forms, direct manipulation); Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Design, Human Factors, Standardization

Keywords:
model-driven development, multimodal interaction, transformational approach, user interface eXtensible markup language