ABSTRACT
Post-task ratings of difficulty in a usability test have the potential to provide diagnostic information and be an additional measure of user satisfaction. But the ratings need to be reliable as well as easy to use for both respondents and researchers. Three one-question rating types were compared in a study with 26 participants who attempted the same five tasks with two software applications. The types were a Likert scale, a Usability Magnitude Estimation (UME) judgment, and a Subjective Mental Effort Question (SMEQ). All three types could distinguish between the applications with 26 participants, but the Likert and SMEQ types were more sensitive with small sample sizes. Both the Likert and SMEQ types were easy to learn and quick to execute. The online version of the SMEQ question was highly correlated with other measures and had equal sensitivity to the Likert question type.
Supplemental Material
Available for Download
- Brooke, J. (1996). SUS: A Quick and Dirty Usability Scale. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester&I.L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor&Francis, 189--194.Google Scholar
- Kirakowski, J.&Cierlik, B. (1998). Measuring the Usability of Web Sites, Proceedings of the Human factors and Ergonomics Society 42nd Annual Meeting, 424--428.Google ScholarCross Ref
- Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability. studies: The ASQ. SIGCHI Bulletin, 23, 1, 78--81. Google ScholarDigital Library
- Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14, 463--488.Google ScholarCross Ref
- McGee, M. (2004). Master Usability Scaling: Magnitude Estimation and Master Scaling Applied to Usability Measurement. Proc. CHI 2004 ACM Press (2004), pp. 335--342 Google ScholarDigital Library
- Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.Google Scholar
- Sauro, J.&Lewis, J.R.(2009)"Correlations among Prototypical Usability Metrics: Evidence for the Construct of Usability"Proc. CHI 2009 In Press. Google ScholarDigital Library
- Tedesco, D.&Tullis, T. (2006). A Comparison of Methods for Eliciting Post-Task Subjective Ratings in Usability Testing. Usability Professionals Association (UPA), 2006, 1--9.Google Scholar
- Tullis, T. and Stetson, J. (2004). A Comparison of Questionnaires for Assessing Website Usability. Usability Professionals Association (UPA), 2004, 7--11.Google Scholar
- Zijlstra, F. (1993). Efficiency in work behavior. A design approach for modern tools. PhD thesis, Delft University of Technology. Delft, The Netherlands: Delft University Press.Google Scholar
- Zijlstra, F.R.H&Doorn, L. van (1985). The construction of a scale to measure subjective effort. Technical Report, Delft University of Technology, Department of Philosophy and Social Sciences.Google Scholar
Index Terms
Comparison of three one-question, post-task usability questionnaires
Recommendations
When designing usability questionnaires, does it hurt to be positive?
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWhen designing questionnaires there is a tradition of including items with both positive and negative wording to minimize acquiescence and extreme response biases. Two disadvantages of this approach are respondents accidentally agreeing with negative ...
UMUX-LITE: when there's no time for the SUS
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsIn this paper we present the UMUX-LITE, a two-item questionnaire based on the Usability Metric for User Experience (UMUX) [6]. The UMUX-LITE items are This system's capabilities meet my requirements and This system is easy to use." Data from two ...
A review of post-study and post-task subjective questionnaires to guide assessment of system usability
Usability is a key consideration when developing an interactive software application because of the various outcomes it can produce. Accordingly, numerous evaluation methods have been proposed, however, a recent review of usability methods concluded ...
Comments