skip to main content
10.1145/1647314.1647347acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Formal description techniques to support the design, construction and evaluation of fusion engines for sure (safe, usable, reliable and evolvable) multimodal interfaces

Published: 02 November 2009 Publication History

Abstract

Representing the behaviour of multimodal interactive systems in a complete, concise and non-ambiguous way is still a challenge for formal description techniques (FDT). Depending on the FDT, multimodal interactive systems feature specific characteristics that are either cumbersome or impossible to capture with classical FDT. This is due to the multiple (potentially synergistic) use of modalities and the strong temporal constraints usually encountered in this kind of systems that have to be dealt with exhaustively if FDT are used. This paper focuses on the requirements for the modelling and construction of fusion engines for multimodal interfaces. It proposes a formal description technique dedicated to the engineering of interactive multimodal systems able to address the challenges of fusion engines. Such benefits are presented on a set of examples illustrating both the constructs and the process.

References

[1]
Barboni E., Conversy S., Navarre D., Palanque P. Model-Based Engineering of Widgets, User Applications and Servers Compliant with ARINC 661 Specification. Design Specification and Verification of Interactive Systems (DSV-IS 2006), Dublin, Ireland, 26/07/06--28/07/06, Springer-Verlag, LNCS, pp. 25--38, 2006.
[2]
Bass, L., Pellegrino, R., Reed, S., Seacord, R., Sheppard, R., and Szezur, M. R. The Arch model: Seeheim revisited. proceeding of the User Interface Developpers' workshop. 91.
[3]
Bastide R., Navarre D., Palanque P., Schyn A., Dragicevic P. A Model-Based Approach for Real-Time Embedded Multimodal Systems in Military Aircrafts. In : ICMI 2004 -- Sixth International Conference on Multimodal Interfaces, ACM Press, pp. 243--250.
[4]
Bastide, R., Navarre, D., and Palanque, P. 2002. A model--based tool for interactive prototyping of highly interactive applications. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20 -- 25, 2002). CHI '02. ACM, New York, NY, 516--517
[5]
Bastide R.&Palanque P. Petri nets with objects for specification, design and validation of user-driven interfaces. In proceedings of the third IFIP conference on Human-Computer Interaction, Interact'90. Cambridge 27--31 August 1990.
[6]
Bernhaupt R., Palanque P., Winckler M., Navarre D. 2007. Usability Study of Multi-Modal Interfaces using Eye-Tracking. Proceedings of INTERACT 2007. LNCS 6643, Springer Verlag.
[7]
Bernhaupt R., Navarre D., Palanque P., Winckler M. 2007. Model-Based Evaluation: A New Way to Support Usability Evaluation of Multimodal Interactive Applications. In "Maturing Usability: Quality in Software, Interaction and Quality" Springer Verlag, April 2007.
[8]
Bolt R. A., "Put-that-there": Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262--270, July 1980
[9]
Bouchet, J., Nigay, L., and Ganille, T. 2004. ICARE software components for rapidly developing multimodal interfaces. In Proceedings of the 6th international Conference on Multimodal interfaces (State College, PA, USA, October 13 -- 15, 2004). ICMI '04. ACM, New York, NY, 251--258.
[10]
Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. 1997. QuickSet: multimodal interaction for distributed applications. In Proceedings of the Fifth ACM international Conference on Multimedia (Seattle, Washington, United States, November 09 -- 13, 1997). MULTIMEDIA '97. ACM, New York, NY, 31--40.
[11]
Coutaz J., Nigay L., Salber D., Blandford A., May J., Young R. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE properties. In Proceedings of the INTERACT'95 conference. pages 115--120. 1995. S. A. Arnesen&D. Gilmore Eds., Chapman&Hall Publ., Lillehammer, Norway.
[12]
Esteban, O., Chatty, S.&Palanque P, 1995, Whizz'Ed: a visual environment for building highly interactive interfaces. Proceedings of the Interact'95 conference. 1995, p121--126.
[13]
Flippo, F., Krebs, A., and Marsic, I. 2003. A framework for rapid development of multimodal interfaces. In Proceedings of the 5th international Conference on Multimodal interfaces (Vancouver, British Columbia, Canada, November 05 -- 07, 2003). ICMI '03. ACM, New York, NY, 109--116.
[14]
Genrich, H. J. 1991. Predicate/Transitions Nets. In High-Levels Petri Nets: Theory and Application . K. Jensen and G. Rozenberg (Eds.)Berlin: Springer Verlag (1991) pp. 3--43.
[15]
Green, M., "A survey of three dialogue models." 1986. ACM Trans. Graph., n°3, Vol. 5, pp. 244--275.
[16]
Harel, D. 1987, "Statecharts: A visual formalism for complex systems." Sci. Comput. Program., Elsevier North--Holland, Inc., 1987, n°3, Vol. 8, pp. 231--274.
[17]
Heymann, Michael and Degani, Asaf., "Formal Analysis and Automatic Generation of User Interfaces: Approach, Methodology, and an Algorithm." Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 49, pp. 311--330.
[18]
Hinckley, K., Czerwinski, M. and Sinclair, M., 1998. Interaction and modeling techniques for desktop two-handed input. In Proceedings of the 11th Annual ACM Symposium on User interface Software and Technology. UIST '98. ACM, New York, NY, 49--58.
[19]
Jacob, R. J., Deligiannidis, L., and Morrison, S. 1999. A software model and specification language for non-WIMP user interfaces. ACM Trans. Comput.-Hum. Interact. 6, 1 (Mar. 1999), 1--46.
[20]
Johnston, M. and Bangalore, S. 2005. Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11, 2 (Jun. 2005), 159--187
[21]
Krahnstoever, N., Kettebekov, S., Yeasin, M., and Sharma, R. 2002. A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays. In Proceedings of the 4th IEEE international Conference on Multimodal interfaces -- Volume 00 (October 14 -- 16, 2002). International Conference on Multimodal Interfaces. IEEE Computer Society, Washington, DC, 349.
[22]
Keh, H. C. and Lewis, T. G., 1991. Direct-manipulation user interface modeling with high-level Petri nets. In Proceedings of the 19th Annual Conference on Computer Science (San Antonio, Texas, United States). CSC '91. ACM, New York, NY, 487--495.
[23]
Lalanne D., Nigay L., Palanque P., Robinson P., Vanderdonckt J., Ladry J-F 2009 Fusion Engines for Multimodal Interfaces: a survey. International Conference on Multimodal Interfaces and Workshop on Machine Learning for Multi-modal Interaction (ICMI--MLMI 2009), Cambridge, Massachusetts, USA, 2009, ACM, this volume.
[24]
Latoschik, M.E., 2002. Designing transition networks for multimodal VR-interactions using a markup language. Multimodal Interfaces, 2002. In Proceedings of the Fourth IEEE International Conference. pp. 411--416, 2002.
[25]
Mellor S. and Balcer M. 2002 Executable UML: A Foundation for Model-Driven Architectures. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2002.
[26]
Navarre D., Palanque P., Bastide R., Schyn A., Winckler M., Nedel L., Freitas C. 2005 A Formal Description of Multimodal Interaction Techniques for Immersive Virtual Reality Applications. In : Human-Computer Interaction -- INTERACT 2005: IFIP TC13 International Conference, Springer-Verlag, pp. 170--185, September 2005.
[27]
Neal, J. G., Thielman, C. Y., Dobes, Z., Haller, S. M., and Shapiro, S. C. 1989. Natural language with integrated deictic and graphic gestures. In Proceedings of the Workshop on Speech and Natural Language (Cape Cod, Massachusetts, 1989). Human Language Technology Conference. Association for Computational Linguistics, Morristown, NJ, 410--423
[28]
Nigay, L. and Coutaz, J. 1995. A generic platform for addressing the multimodal challenge. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'95). ACM Press/Addison-Wesley Publishing Co., New York, NY, 98--105.
[29]
Object Management Group: Unified Modelling Language (UML) 2.0 Superstructure Specification, August 2003. Ptc/03-08-02, pp. 455--510
[30]
Oviatt, S. Ten myths of Multimodal Interaction Communication of the ACM; 42: 11: 74--81, 1999.
[31]
Palanque P., Ladry J-F, Navarre D.&Barboni E. High-Fidelity Prototyping of Interactive Systems can be Formal too 13th Int. Conf. on Human-Computer Interaction (HCI International 2009) San Diego, CA, USA, Springer Verlag, LNCS 5610.
[32]
Parnas, D. L., 1969. On the use of transition diagrams in the design of a user interface for an interactive computer system. In Proceedings 24th National ACM Conference. pp. 379--385. 1969
[33]
Sun, Y., Chen, F., Shi, Y., and Chung, V. 2006. A novel method for multi-sensory data fusion in multimodal human computer interaction. In Proceedings of the 18th Australia Conference on Computer-Human interaction. OZCHI '06, vol. 206. ACM, New York, NY, 401--404.
[34]
Willans, J. S. and Harrison, M. D., 2001. Prototyping Pre-implementation Designs of Virtual Environment Behaviour. In Proceedings of the 8th IFIP international Conference on Engineering For Human-Computer interaction (May 11 -- 13, 2001). LNCS, vol. 2254. Springer-Verlag, 91--108.
[35]
Winckler, M.; Palanque, P. 2003. StateWebCharts: a Formal Description Technique Dedicated to Navigation Modelling of Web Applications. International Workshop on Design, Specification and Verification of Interactive Systems -- (DSVIS'2003), LNCS n° 2669

Cited By

View all
  • (2024)A Systematic Process to Engineer Dependable Integration of Frame-based Input Devices in a Multimodal Input Chain: Application to Rehabilitation in HealthcareProceedings of the ACM on Human-Computer Interaction10.1145/36646338:EICS(1-31)Online publication date: 17-Jun-2024
  • (2021)Dependability and Safety: Two Clouds in the Blue Sky of Multimodal InteractionProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479881(781-787)Online publication date: 18-Oct-2021
  • (2020)Model-Based Testing of Post-WIMP Interactions Using Object Oriented Petri-NetsFormal Methods. FM 2019 International Workshops10.1007/978-3-030-54994-7_35(486-502)Online publication date: 13-Aug-2020
  • Show More Cited By

Index Terms

  1. Formal description techniques to support the design, construction and evaluation of fusion engines for sure (safe, usable, reliable and evolvable) multimodal interfaces

            Recommendations

            Comments

            Information & Contributors

            Information

            Published In

            cover image ACM Conferences
            ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces
            November 2009
            374 pages
            ISBN:9781605587721
            DOI:10.1145/1647314
            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

            Sponsors

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            Published: 02 November 2009

            Permissions

            Request permissions for this article.

            Check for updates

            Author Tags

            1. formal description techniques
            2. fusion engines
            3. interactive software engineering
            4. model-based approaches
            5. multimodal interfaces
            6. safety-critical interactive systems

            Qualifiers

            • Research-article

            Conference

            ICMI-MLMI '09
            Sponsor:

            Acceptance Rates

            Overall Acceptance Rate 453 of 1,080 submissions, 42%

            Contributors

            Other Metrics

            Bibliometrics & Citations

            Bibliometrics

            Article Metrics

            • Downloads (Last 12 months)6
            • Downloads (Last 6 weeks)0
            Reflects downloads up to 17 Feb 2025

            Other Metrics

            Citations

            Cited By

            View all
            • (2024)A Systematic Process to Engineer Dependable Integration of Frame-based Input Devices in a Multimodal Input Chain: Application to Rehabilitation in HealthcareProceedings of the ACM on Human-Computer Interaction10.1145/36646338:EICS(1-31)Online publication date: 17-Jun-2024
            • (2021)Dependability and Safety: Two Clouds in the Blue Sky of Multimodal InteractionProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479881(781-787)Online publication date: 18-Oct-2021
            • (2020)Model-Based Testing of Post-WIMP Interactions Using Object Oriented Petri-NetsFormal Methods. FM 2019 International Workshops10.1007/978-3-030-54994-7_35(486-502)Online publication date: 13-Aug-2020
            • (2019)Deep System Knowledge Required: Revisiting UCD Contribution in the Design of Complex Command and Control SystemsHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29381-9_42(699-720)Online publication date: 23-Aug-2019
            • (2018)MIODMIT: A Generic Architecture for Dynamic Multimodal Interactive SystemsHuman-Centered Software Engineering10.1007/978-3-030-05909-5_7(109-129)Online publication date: 31-Dec-2018
            • (2015)Dependable multi-touch interactions in safety critical industrial contexts: Application to aeronautics2015 IEEE 13th International Conference on Industrial Informatics (INDIN)10.1109/INDIN.2015.7281868(980-987)Online publication date: Jul-2015
            • (2014)A fault-tolerant architecture for resilient interactive systemsProceedings of the 26th Conference on l'Interaction Homme-Machine10.1145/2670444.2670462(80-90)Online publication date: 28-Oct-2014
            • (2014)Multi-Touch interactions for control and display in interactive cockpitsProceedings of the International Conference on Human-Computer Interaction in Aerospace10.1145/2669592.2669650(1-10)Online publication date: 30-Jul-2014
            • (2014)Formal modelling of dynamic instantiation of input devices and interaction techniquesProceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems10.1145/2607023.2610286(173-178)Online publication date: 17-Jun-2014
            • (2011)Formal tasks and systems models as a tool for specifying and assessing automation designsProceedings of the 1st International Conference on Application and Theory of Automation in Command and Control Systems10.5555/2248467.2248474(50-59)Online publication date: 26-May-2011
            • Show More Cited By

            View Options

            Login options

            View options

            PDF

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            Figures

            Tables

            Media

            Share

            Share

            Share this Publication link

            Share on social media