skip to main content
10.1145/1124772.1124867acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Article

Automated summative usability studies: an empirical evaluation

Published: 22 April 2006 Publication History

Abstract

This paper evaluates a method for summative usability testing using an automated data collection system. We found automated summative testing to be a simple and effective alternative to lab-based summative testing and could be successfully conducted remotely. In our study, a web-based control window led participants through the summative study, provided tasks to perform, and asked follow up questions about the user experience. Using a within-group comparison, we found no major differences between data collected by a usability engineer and that collected through an automated testing system for performance metrics. Using a between-group comparison, we found automated summative studies could be conducted remotely with minor but acceptable differences in time on task and likelihood to give up on a task compared to lab-based testing. Task success and task satisfaction ratings were not different between remote and lab-based summative testing. Written comments provided by participants through the testing system were sufficient to identify the major usability problems that led to task failure but did not reveal as comprehensive a set of issues as did a usability engineer observing the sessions.

References

[1]
American National Standards Institute (2001). Common industry format for usability test reports (ANSI-NCITS 354-2001). American National Standards Institute.
[2]
Bernheim Brush, A. J., Ames, M., & Davis, J. (2004). A Comparison of Synchronous Remote and Local Usability Studies for an Expert Interface. In Ext. Abstracts CHI 2004, ACM Press, 1179--1182.
[3]
Dray, S., and Siegel, D. (2002). Remote possibilities? International usability testing at a distance. Interactions, ACM Press, 10--17.
[4]
International Organization for Standards (1998). Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability (ISO 9241-11:1998(E)). International Organization for Standards.
[5]
Molich, R., Ede M. R., Kaasgaard, K., Karyukin, K. (2004). Comparative usability evaluation. Behaviour & Information Technology, 23, 65--74.
[6]
Molich, R., Thomsen, A. D., Karyukina, B., Schmidt, L., Ede, M., van Oel, W., and Arcuri, M. (1999). Comparative evaluation of usability tests. In Ext. Abstracts CHI 1999, ACM Press.
[7]
Thompson, K. E., Rozanski, E. P., & Haake, A. R. (2004). Here, There, Anywhere: Remote Usability Testing That Works. In Proc. SIGITE 2004, ACM Press, 132--137.
[8]
Tullis, T., Fleischman, S., McNulty, M., Cianchette, C., and Bergel, M. (2002). An empirical comparison of lab and remote usability testing of web Sites. In Proc. UPA 2002, Usability Professional's Association.
[9]
Waterson, S., Landay, J, and Matthews, T. (2002). In the lab and out in the wild: Remote usability testing for mobile devices. In Ext. Abstracts CHI 2002, ACM Press, 796--797.

Cited By

View all
  • (2023)Asynchronous Remote Usability Tests Using Web-Based Tools Versus Laboratory Usability Tests: An Experimental StudyIEEE Transactions on Human-Machine Systems10.1109/THMS.2023.328222553:4(731-742)Online publication date: Aug-2023
  • (2022)Welfare Through Competence: A Framework for Animal-Centric Technology DesignFrontiers in Veterinary Science10.3389/fvets.2022.8859739Online publication date: 30-Jun-2022
  • (2021)WEBSITE DESIGN AND EVALUATIONHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch39(1016-1036)Online publication date: 13-Aug-2021
  • Show More Cited By

Index Terms

  1. Automated summative usability studies: an empirical evaluation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2006
    1353 pages
    ISBN:1595933727
    DOI:10.1145/1124772
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 April 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. automated testing
    2. empirical methods
    3. remote testing
    4. summative testing
    5. usability methods

    Qualifiers

    • Article

    Conference

    CHI06
    Sponsor:
    CHI06: CHI 2006 Conference on Human Factors in Computing Systems
    April 22 - 27, 2006
    Québec, Montréal, Canada

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)25
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Asynchronous Remote Usability Tests Using Web-Based Tools Versus Laboratory Usability Tests: An Experimental StudyIEEE Transactions on Human-Machine Systems10.1109/THMS.2023.328222553:4(731-742)Online publication date: Aug-2023
    • (2022)Welfare Through Competence: A Framework for Animal-Centric Technology DesignFrontiers in Veterinary Science10.3389/fvets.2022.8859739Online publication date: 30-Jun-2022
    • (2021)WEBSITE DESIGN AND EVALUATIONHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch39(1016-1036)Online publication date: 13-Aug-2021
    • (2021)USABILITY AND USER EXPERIENCE: DESIGN AND EVALUATIONHANDBOOK OF HUMAN FACTORS AND ERGONOMICS10.1002/9781119636113.ch38(972-1015)Online publication date: 13-Aug-2021
    • (2019)Automating Mockup-Based Usability Testing on the Mobile DeviceGreen, Pervasive, and Cloud Computing10.1007/978-3-030-19223-5_10(128-143)Online publication date: 26-May-2019
    • (2018)An Automatic User Activity Analysis Method for Discovering Latent Requirements: Usability Issue Detection on Mobile ApplicationsSensors10.3390/s1809296318:9(2963)Online publication date: 5-Sep-2018
    • (2017)A systematic review of software usability studiesInternational Journal of Information Technology10.1007/s41870-017-0048-1Online publication date: 11-Dec-2017
    • (2015)Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability StudyInternational Journal of Human-Computer Interaction10.1080/10447318.2015.106466531:8(545-553)Online publication date: 25-Jun-2015
    • (2014)Conceptualizing Quality in Software IndustryGSTF Journal on Computing (JoC)10.7603/s40601-013-0027-83:3Online publication date: 27-Feb-2014
    • (2013)MATECHI '13 Extended Abstracts on Human Factors in Computing Systems10.1145/2468356.2468404(265-270)Online publication date: 27-Apr-2013
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media