Jeff Sauro
Erika Kindlund
ABSTRACT
Current methods to represent system or task usability in a single metric do not include all the ANSI and ISO defined usability aspects: effectiveness, efficiency & satisfaction. We propose a method to simplify all the ANSI and ISO aspects of usability into a single, standardized and summated usability metric (SUM). In four data sets, totaling 1860 task observations, we show that these aspects of usability are correlated and equally weighted and present a quantitative model for usability. Using standardization techniques from Six Sigma, we propose a scalable process for standardizing disparate usability metrics and show how Principal Components Analysis can be used to establish appropriate weighting for a summated model. SUM provides one continuous variable for summative usability evaluations that can be used in regression analysis, hypothesis testing and usability reporting.
- Abran, A., Surya, W., Khelifi, A., Rilling, J., Seffah, A., Robert, F. (2003). Consolidating the ISO Usability Models. Paper presented at 11th annual International Software Quality Management Conference.Google Scholar
- ANSI (2001). Common industry format for usability test reports (ANSI-NCITS 354-2001). Washington, DC: American National Standards Institute.Google Scholar
- Babiker, E.M., Fujihara, H., Boyle, Craig. D. B. (1991). A metric for hypertext usability. In Proc. 11th Annual International Conference on Systems documentation, (pp.95--104). ACM Press. Google ScholarDigital Library
- Breyfogle, F. (1999). Implementing Six Sigma: Smarter Solutions Using Statistical Methods. John Wiley and Sons.Google Scholar
- Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P. Jordan, B. Thomas, and B. Weerdmeester (Eds.), Usability Evaluation in Industry (pp.189--194). London: Taylor and Francis. See also http://www.cee.hw.ac.uk/~ph/sus.htmlGoogle Scholar
- Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245--276.Google Scholar
- Chin, J. P., Diehl, V. A., and Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proc. CHI '88. (pp. 213--218). Washington, D.C.: ACM Press. See also http://www.lap.umd.edu/QUIS/index.html Google ScholarDigital Library
- Cordes, R. E (1984). Application of Magnitude Estimation for Evaluating Software Ease of Use. In Gavriel Salvendy (Ed.) First USA-Japan Conference on Human Computer Interaction, Amsterdam: Elsevier Science Publishers.Google Scholar
- Dumas, J. S. (2003). User-based evaluations. In J. A. Jacko and A. Sears (Eds.), The Human-Computer Interaction Handbook (pp. 1093--1117). Mahwah, NJ: Lawrence Erlbaum. Google ScholarDigital Library
- Dunteman, George H, (1989) Principal Components Analysis. In Sage University Papers Series Quantitative Applications in the Social Sciences; No. 07-069 Newbury Park Sage Publications, Inc.Google Scholar
- Dumas, J., and Redish, J. C. (1999). A practical guide to usability testing. Portland, OR: Intellect. Google ScholarDigital Library
- Frøkjær, E., Hertzum, M., and Hornbæk, K. (2000) Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In Proc. CHI 2000, (pp.345--352). Washington, D.C.: ACM Press. Google ScholarDigital Library
- Gliem, J. and Gliem, R. (2003). Calculating, Interpreting, and Reporting Cronbach's Alpha Reliability Coefficient for Likert-Type Scales. In 2003 Midwest Research to Practice Conference in Adult, Continuing and Community Education. Columbus, OH.Google Scholar
- Hair, Anderson, Tatham, Black (1998) Multivariate Data Analysis Fifth Edition. NJ: Prentice Hall. Google ScholarDigital Library
- Hagen, E. (Ed.) (1976). Human Reliability Analysis, control and instrumentation. Nuclear Safety. 17(3), 315--326.Google Scholar
- Hassenzahl, M. Sandweg, N. (2004). From Mental Effort to Perceived Usability: Transforming Experiences into Summary Assessments. In the Extended Abstracts of the 2004 conference on Human Factors and Computing Systems (pp 1283--1286). Google ScholarDigital Library
- Harry, M. J (1987). The Nature of Six Sigma Quality. Technical Report, Government Electronics Group, Motorola Inc. Scottsdale, AZ.Google Scholar
- ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva, Switzerland: Author.Google Scholar
- Jolliffe, I. T. (1972). Discarding variables in a principal component analysis 1: Artificial data. Applied Statistics, 21, 160--173.Google ScholarCross Ref
- Jolliffe, Ian T.(2002). Principal Component Analysis. Secaucus, NJ, USA: Springer-Verlag.Google Scholar
- Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational Psychology Measurement, 20, 141--151.Google ScholarCross Ref
- Kirakowski, J. (1996). The Software Usability Measurement Inventory: Background and usage. In P. Jordan, B. Thomas, and B. Weerdmeester (Eds.), Usability Evaluation in Industry (pp. 169--178). London, UK: Taylor and Francis. (Also, see http://www.ucc.ie/hfrg/questionnaires/sumi/index.html).Google Scholar
- Kirakowski, J., and Corbett, M. (1993). SUMI: The Software Usability Measurement Inventory. British Journal of Educational Technology, 24, 210--212.Google ScholarCross Ref
- Lewis, J. R (1982) "Testing Small System Customer Setup" in Proceedings of the Human Factors Society 26th Annual Meeting p. 718--720.Google Scholar
- Lewis, J (1991) A Rank-Based Method for the Usability Comparison of Competing Products. In Proceedings of the Human Factors and Ergonomics Society 35th Annual Meeting San Francisco California (pp1312--1316).Google Scholar
- Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. SIGCHI Bulletin, 23, 78--81. Google ScholarDigital Library
- Lewis, J. R. (1992). Psychometric evaluation of the Post-Study System Usability Questionnaire: The PSSUQ. In Proceedings of the Human Factors Society 36th Annual Meeting (pp. 1259--1263). Atlanta, GA: Human Factors Society.Google ScholarCross Ref
- Lewis, J. R. (1993). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use (Tech. Report 54.786). Boca Raton, FL: IBM Corp. http://drjim.0catch.com/usabqtr.pdfGoogle Scholar
- Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57--78. Google ScholarDigital Library
- Lewis, J.R. (In Press). Handbook of Human Factors and Ergonomics 3rd Edition. Gavriel Salvendy Editor. John Wiley & Sons.Google Scholar
- Martin,P and Bateson, P (1993) Measuring Behaviour. (2nd Edition) Cambridge {England}; New York, NY, USA: Cambridge University Press.Google Scholar
- McGee, M. (2003). Usability magnitude estimation. Proc. HFES, 47th Annual Meeting, (691--695).Google ScholarCross Ref
- McGee, M (2004). Master usability scaling:magnitude estimation and master scaling applied to usability measurement. In Proc. CHI 2004, (pp 335--342). Washington, D.C.: ACM Press. Google ScholarDigital Library
- McIver, J. P., & Carmines, E. G. (1981). Unidimensional scaling. Thousand Oaks, CA: Sage.Google Scholar
- Molich, R., Ede, M., Kaasgaard, K., and Karyukin, B (2004). Comparative Usability Evaluation. Behaviour & Information Technology, 23(1), 65--74. Google ScholarDigital Library
- Morrison, D. F. (1976). Multivariate statistical methods (2nd ed.). New York: McGraw-Hill.Google Scholar
- Nielsen, J (1994) Usability Engineering. San Francisco: Morgan Kaufman. Google ScholarDigital Library
- Nielsen, J. and Levy, J. (1994) Measuring Usability: Preference vs. Performance. Communications of the ACM, 37, p. 66--76. Google ScholarDigital Library
- Nielsen, J. (1997). Usability testing. In G. Salvendy (Ed.), The Handbook of Human Factors and Ergonomics, (2rd Edition). John Wiley & Sons.Google Scholar
- Nunnally, J. C. (1978). Psychometric Theory. New York, NY: McGraw-Hill.Google Scholar
- Park, Kyung S. (1997). Human Error. In Gavriel Salvendy (Ed.), The Handbook of Human Factors and Ergonomics, (2rd Edition). John Wiley & Sons.Google Scholar
- Sauro, J. (2004) How long should a task take? Identifying Spec Limits for Task Times in Usability Tests. Retrieved September 13, 2004, from Measuring Usability Web site : http://measuringusability.com/time_specs.htmGoogle Scholar
- Sauro, J. (2004) How Do You Calculate a Z-Score? Retrieved September 13, 2004, from Measuring Usability Web site: http://measuringusability.com/z_calc.htmGoogle Scholar
- Sauro, J & Kindlund E. (In Press) Making Sense of Usability Metrics: Usability and Six Sigma, in Proceedings of the 14th Annual Conference of the Usability Professionals Association, Montreal, Canada.Google Scholar
- Stevens, S.S. (1975). Psychophysics: Introduction to its Perceptual, Neural, and Social Prospects. New York: John Wiley.Google Scholar
- Whiteside, J., Bennett, J. and Holtzblatt, K. (1988) "Usability Engineering: Our Experience and Evolution" in The Handbook of Human Computer Interaction Elsevier Science Publishers, Amsterdam, pp 791--817.Google Scholar
Index Terms
- A method to standardize usability metrics into a single score
Recommendations
Correlations among prototypical usability metrics: evidence for the construct of usability
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsCorrelations between prototypical usability metrics from 90 distinct usability tests were strong when measured at the task-level (r between .44 and .60). Using test-level satisfaction ratings instead of task-level ratings attenuated the correlations (r ...
Usability measurement and metrics: A consolidated model
Usability is increasingly recognized as an important quality factor for interactive software systems, including traditional GUIs-style applications, Web sites, and the large variety of mobile and PDA interactive services. Unusable user interfaces are ...
New ISO Standards for Usability, Usability Reports and Usability Measures
Proceedings, Part I, of the 18th International Conference on Human-Computer Interaction. Theory, Design, Development and Practice - Volume 9731Several new and revised ISO standards will be published in 2016/17 that define the basic terms and concepts of usability ISO 9241-11, give guidance on processes and outcomes of human-centred design ISO 9241-220, provide examples of measures that can be ...
Comments