skip to main content
10.1145/1180995.1181063acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
Article

Evaluating usability based on multimodal information: an empirical study

Published: 02 November 2006 Publication History

Abstract

New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multiple information streams in usability evaluation. In the study, easy, medium and difficult versions of a game task were used to vary the levels of mental effort. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Four findings indicate the potential value of usability evaluations based on multimodal information: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different mental workload levels appeared to correlate with eye movement patterns, especially with a combined eye-hand movement measure. Third, HRV showed correlations with saccade speed. Finally, we present a new method using the ratio of eye fixations over mouse clicks to evaluate performance in more detail. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.

References

[1]
Hollnagel, E. From human factors to cognitive systems engineering: Human-machine interactions in the 21st century. In Kitamura, M., and Kimura, I. (eds.), Anzen-no-Tankyu (Researches on Safety). ERCPublishing, Tokyo, 2003.
[2]
Andreassi, J. L. Psychophysiology: Human Behavior and Physiological Response, 4th Edition. Lawrence Erlbaum Associates, Mahwah, NJ, 2000.
[3]
Picard, R. W. Affective Computing. MIT Press, Cambridge, MA, 1997.
[4]
Kramer, A. F. Physiological metrics of mental workload: A review of recent progress. In Damos, D. L. (ed.), Multiple-Task Performance, Taylor and Francis, London, 1991, 279--328.
[5]
Rowe, D. W., Sibert, J. and Irwin, D. Heart rate variability: Indicator of user state as an aid to human-computer interaction. In Proceeding of CHI1998, 480--487. 1998.
[6]
Takahashi, K., Nakayama, M. and Shimizu, Y. The response of eye-movement and pupil size to audio instruction while viewing a moving target. In Proceedings of the Eye Tracking Research & Applications 2000.
[7]
Hopman, J. C. W., Kollee, L. A. A., Stoelinga, G. B. A., van Geijn, H. P. and van Ravenswaaij-Arts, C. M. A. Heart rate variability. Annals of Internal Medicine 1993, 118, 436--447.
[8]
Malik, M. Heart rate variability. Circulation 1996, 93, 1043--1065.
[9]
Pomeranz, B., and Macaulay, R. J. B., Caudill, M. A., et al. Assessment of autonomic function in humans by heart rate spectral analysis. Am J Physiol Heart Circ Physiol 1985, 248, 151--H153.
[10]
Pagani, M., Lombardi, F., Guzzetti, S., et al. Power spectral analysis of heart rate and arterial pressure variabilities as a maker of sympatho-vagal interaction in man and conscious dog. Cir Res 1986, 59, 178--93.
[11]
Ettema, J. H. and Ziclhuis, R. L. Physiological parameters of mental load. Ergonomics 1971, 14, 137--44.
[12]
Luczak, I. I. and Lauring, W. J. An analysis of heart rate variability. Ergonomics 1973, 16, 85--97.
[13]
Hyndman, B. W., and Gregory, J. R. Spectral analysis of sinus arrhythmia during mental loading. Ergonomics 1975, 18, 255--70.
[14]
Boutcher, S. H., Naugent, F. W., and Mclaren, P. F., et al. Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology 1998, 35, 16--22.
[15]
Hyde, C., and Izard, C. E. Cardiac rhythmicities and attention in young children. Psychophysiology 1997, 34, 547--52.
[16]
Vicente, K. J., Thornton, C., and Moray, N. Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors 1987, 29, 171--182.
[17]
Tattersall, A. J., and Hockey, G. R. J. Level of operator control and changes in heart rate variability during simulated flight maintenance. Human Factors 1995, 37, 682--698.
[18]
Hess, E. H. and Polt, J. M. Pupil size in relation to mental activity during simple problem solving. Science 1964, 132, 1190--1192.
[19]
Hoecks, B., and Levelt, W. Pupillary dilation as a measure of attention: A quantitative system analysis. Behavior Research Methods, Instruments, & Computers 1993, 25, 16--26.
[20]
Juris, M. and Velden, M. The pupillary response to mental overload. Physiological Psychology 1977, 5 (4), 421--424.
[21]
Kahneman, D. Pupillary responses in a pitch-discrimination task. Perception & Psychophysics 1967, 2, 101--105.
[22]
Nakayama, M., and Takahashi, K. The act of task difficulty and eye-movement frequency for the oculo-motor indices. In Proceedings of the Eye Tracking Research and Application 2002, 37--42.
[23]
Beatty, J. Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin 1982, 91 (2), 276--292.
[24]
Iqbal, S. T., Zheng, X. S., and Bailey, B. P. Task-evoked pupillary response to mental workload in human-computer interaction. In Proceeding of CHI 2004, ACM Press, 1477--1480.
[25]
Hess, E. H., and Petrovich, S. B. Pupillary behavior in communication. In Siegman, A. W., and Feldstein, S. (eds.), Nonverbal Behavior and Communication. Erlbaum, Hillsdale, NJ, 1987, 327--348.
[26]
Rayner, K. Eye movements and information processing: 20 years of research. Psychological Bulletin 1998, 124 (3), 372--422.
[27]
Noton, D., and Stark, L. W. Scanpath in saccadic eye movements while viewing and recognizing patterns. Vision Research 1971a, 11, 929--942.
[28]
Goldberg, H., and Kotval, X. P. Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 1999, 24, 631--645.
[29]
Goldberg, J. H., Stimson, MJ, Lewenstein, M., Scott, N. and Wichansky, A. M. Eye tracking in web search tasks: Design implications. In Proceedings of the Symposium on ETRA 2002, March 2002.
[30]
Ellis, S., Candera, R., Misner, J., C. S., Lankford, C. P., and Hutshinson, T. E. Windows to the soul? What eye movements tell us about software usability? In Proceedings of the Usability Professionals' Association Conference, 1998, 151--178.
[31]
Cowen, L., Ball, L. J., and Delin, J. An eye movement analysis of web page usability. In Proceedings of the Annual HCI, 2002, 317--335.
[32]
Altonen, A. Hyrskykari, A., and Räihä, K. 101 spots, or how do users read menus? In Proceedings of CHI 98 Human Factors in Computing Systems, ACM Press, 1998, 132--139.
[33]
Crosby, M. E., and Peterson, W. W. Using eye movements to classify search strategies. In Proceedings of the Human Factors Society 35th Annual Meeting. 1991, 1476--1480.
[34]
Andrews, T., and Coppola, D. Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments. Vision Research 1999, 39, 2947--2953.
[35]
Lin, Y., Zhang, W. J., and Watson, L. G. Using eye movement parameters for evaluating human-machine interface frameworks under normal control operation and fault detection situation. International Journal of Human-Computer Studies 2003, 59 (6), 837--873.
[36]
Wickens, C. D., and Holland, J. G., Engineering Psychology and Human Performance, 3rd Edition. Longman, London, 2000.
[37]
Hébert. a,b., Renée. a,b., Odrée Dionne-Fournelle. a., Martine Crê te. a. and S. J. Lupien. Physiological stress response to video game playing: The contribution of built-in music. Life Sciences 2005, 76, 2371--2380.
[38]
Rubin, J. Handbook of Usability Testing. Wiley and Sons, New York, 1994.
[39]
Hart, S. G., and Staveland, L. E. Development of NASA-TLX (task load index): Results of experimental and theoretical research. In Hancock, P.A., and Meshakati, N. (eds.), Human Mental Workload. North-Holland, Amsterdam, 1988, 39--183.
[40]
Marshall, C., and Rossman, G. B. Designing Qualitative Research, 3rd Edition, Sage, Thousand Oaks, CA, 1999.
[41]
Wilson, G. M., and Sasse, M. A. Investigating the impact of audio degradations on users: Subjective vs. objective assessment methods. In Proceedings of OZCHI 2000: Interfacing Reality in the New Millennium, 2000, 135--142.
[42]
Wilson, G. M. Psychophysiological indicators of the impact of media quality on users. In Proceedings of CHI 2001 Doctoral Consortium, ACM Press, 2001, 95--96.
[43]
Ekman, P., Levenson, R. W., and Friesen, W. V. Autonomic nervous system activity distinguishes among emotions. Science 1983, 221 (4616), 1208--1210.
[44]
Hair, J. F., Anderson, R. E., Tatham, R. L., and Black, W. C. Multivariate Data Analysis, 5th Edition, Prentice-Hall, Englewood Cliffs, NJ, 1998.
[45]
Lin, T., Hu, W. H., Omata, M., and Imamiya, A. Do physiological data relate to traditional usability indexes? In Proceedings of the Australian Conference on Computer Human Interaction (OZCHI'05), 2005.
[46]
Meehan, M., Insko, B., Whitton, M., and Brooks, F. Physiological measures of presence in stressful virtual environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, 2002.
[47]
Lin, Y., Zhang, W. J. and Koubek, R. J. Effective Attention Allocation Behavior and its Measurement: A Preliminary Study, Interacting with Computers, 2004 Vol. 16, Issue 6, pp. 1195--1210.

Cited By

View all
  • (2021)VREEDProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34950025:4(1-20)Online publication date: 30-Dec-2021
  • (2017)Quality Assessment of Interaction Techniques in Immersive Virtual Environments Using Physiological Measures2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC)10.1109/COMPSAC.2017.196(109-117)Online publication date: Jul-2017
  • (2016)The State-of-The-ArtRobust Multimodal Cognitive Load Measurement10.1007/978-3-319-31700-7_2(13-32)Online publication date: 15-Jun-2016
  • Show More Cited By

Index Terms

  1. Evaluating usability based on multimodal information: an empirical study

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces
    November 2006
    404 pages
    ISBN:159593541X
    DOI:10.1145/1180995
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 November 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. eye tracking
    2. multimodal
    3. physiological measures
    4. usability

    Qualifiers

    • Article

    Conference

    ICMI06
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 453 of 1,080 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)23
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)VREEDProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34950025:4(1-20)Online publication date: 30-Dec-2021
    • (2017)Quality Assessment of Interaction Techniques in Immersive Virtual Environments Using Physiological Measures2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC)10.1109/COMPSAC.2017.196(109-117)Online publication date: Jul-2017
    • (2016)The State-of-The-ArtRobust Multimodal Cognitive Load Measurement10.1007/978-3-319-31700-7_2(13-32)Online publication date: 15-Jun-2016
    • (2015)Event-Related Assessment of Hypermedia-Based E-Learning Materials With an HRVBased Method That Considers Individual Differences in UsersInternational Journal of Occupational Safety and Ergonomics10.1080/10803548.2011.1107688217:2(119-127)Online publication date: 8-Jan-2015
    • (2015)Improved Temporal Resolution Heart Rate Variability Monitoring—Pilot Results of Non-Laboratory Experiments Targeting Future Assessment of Human-Computer InteractionInternational Journal of Occupational Safety and Ergonomics10.1080/10803548.2011.1107688117:2(105-117)Online publication date: 8-Jan-2015
    • (2014)Analyzing the benefits of integrative multi-dimensional assessments of usability features in interaction-centered user studiesProceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare10.4108/icst.pervasivehealth.2014.255142(227-230)Online publication date: 20-May-2014
    • (2013)A review of eye-tracking applications as tools for trainingCognition, Technology and Work10.1007/s10111-012-0234-715:3(313-327)Online publication date: 1-Aug-2013
    • (2012)A Video Game Level Analysis Model ProposalProceedings of the 2012 16th International Conference on Information Visualisation10.1109/IV.2012.82(474-479)Online publication date: 11-Jul-2012
    • (2012)Is the Heart Period Variability-based usability measurement of mental effort robust enough against arm movements?2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom)10.1109/CogInfoCom.2012.6422050(601-604)Online publication date: Dec-2012
    • (2011)Identifying Relationships between Physiological Measures and Evaluation Metrics for 3D Interaction TechniquesHuman-Computer Interaction – INTERACT 201110.1007/978-3-642-23765-2_45(662-679)Online publication date: 2011
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media