Accounting for diversity in subjective judgments

ABSTRACT

In this paper we argue against averaging as a common practice in the analysis of subjective attribute judgments, both across and within subjects. Previous work has raised awareness of the diversity between individuals' perceptions. In this paper it will furthermore become apparent that such diversity can also exist within a single individual, in the sense that different attribute judgments from a subject may reveal different, complementary, views. A Multi-Dimensional Scaling approach that accounts for the diverse views on a set of stimuli is proposed and its added value is illustrated using published data. We will illustrate that the averaging analysis provides insight to only 1/6th of the total number of attributes in the example dataset. The proposed approach accounts for more than double the information obtained from the average model, and provides richer and semantically diverse views on the set of stimuli.

References

1. Al-Azzawi, A., Frohlich, D., and Wilson, M., Beauty constructs for MP3 players. CoDesign, 2007. 3(1 supp 1): p. 59 -- 74.
2. Bech-Larsen, T. and Nielsen, N.A., A comparison of five elicitation techniques for elicitation of attributes of low involvement products. Journal of Economic Psychology, 1999. 20: p. 315--341.
3. Breivik, E. and Supphellen, M., Elicitation of product attributes in an evaluation context: A comparison of three elicitation techniques. Journal of Economic Psychology, 2003. 24: p. 77--98.
4. Burnham, K.P. and Anderson, D.R., Multimodel Inference: Understanding AIC and BIC in Model Selection. Sociological Methods&Research, 2004. 33(2): p. 261.
5. Collins, A.M. and Loftus, E.F., A spreading-activation theory of semantic processing. Psychological Review, 1975. 82(6): p. 407--428.
6. Davis, S.B. and Carini, C., Constructing a Player-Centred Definition of Fun for Video Games Design. HCI 2004 Conference, 2004: p. 117--132.
7. Draper, N.R. and Smith, H., Applied Regression Analysis. 1998, New Yoek: John Wiley and Sons Inc.
8. Egger, F.N., " Trust me, I'm an online vendor": towards a model of trust for e--commerce system design. Conference on Human Factors in Computing Systems, 2000: p. 101--102.
9. Fällman, D., In Romance with the Materials of Mobile Interaction: A Phenomenological Approach to the Design of Mobile Information Technology. 2003: Univ.
10. Fransella, F., Bell, R., and Bannister, D., A Manual for Repertory Grid Technique 2003: Wiley.
11. Frøkjær, E., Hertzum, M., and Hornbæk, K., Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, in Proceedings of the SIGCHI conference on Human factors in computing systems. 2000, ACM Press: The Hague, The Netherlands.
12. Green, P.E., Carmone Jr., F.J., and Smith, S.M., Multi-dimensional Scaling, Concepts and Applications. 1989, Boston, MA: Allyn&Bacon.
13. Hassenzahl, M. and Wessler, R., Capturing design space from a user perspective: The Repertory Grid Technique revisited. International Journal of Human-Computer Interaction, 2000. 12(3-4): p. 441--459.
14. Hassenzahl, M. and Trautmann, T., Analysis of web sites with the repertory grid technique, in CHI '01 2001, ACM Press: Seattle, Washington.
15. Hassenzahl, M., Character Grid: a Simple Repertory Grid Technique for Web Site Analysis and Evaluation, in Human Factors and Web Development. Lawrence Erlbaum, J. Ratner, Editor. 2002: Mahwah, NJ. p. 183--206.
16. Hassenzahl, M., The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 2004. 19(4): p. 319--349.
17. Hassenzahl, M. and Sandweg, N., From mental effort to perceived usability: transforming experiences into summary assessments, in CHI '04 extended abstracts on Human factors in computing systems. 2004, ACM Press: Vienna, Austria.
18. Hassenzahl, M. and Tractinsky, N., User experience - a research agenda. Behaviour&Information Technology, 2006. 25(2): p. 91--97.
19. Hassenzahl, M., Aesthetics in interactive products: correlates and consequences of beauty, in Product Experience, Elsevier, Amsterdam, H.N.J. Schifferstein and P. Hekkert, Editors. 2007.
20. Hassenzahl, M. and Ullrich, D., To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers, 2007. 19: p. 429--437.
21. Heidecker, S. and Hassenzahl, M., Eine gruppenspezifische Repertory Grid Analyse der wahrgenommenen Attraktivität von Universitätswebsites, in Mensch&Computer 2007: Konferenz füür interative und kooperative Medien, T.Gross, Editor. 2007: Oldenbourg, Munich. p. 129--138.
22. Hinkin, T.R., A Review of Scale Development Practices in the Study of Organizations. Journal of Management, 1995. 21(5): p. 967.
23. Jordan, P.W., Designing Pleasurable Products: An Introduction to New Human Factors. 2000, London: Taylor&Francis.
24. Karapanos, E. and Martens, J.-B., Characterizing the Diversity in Users' Perceptions, in Human-Computer Interaction - INTERACT 2007. 2007, Springer. p. 515--518.
25. Karapanos, E., Hassenzahl, M., and Martens, J.-B., User experience over time, in CHI '08 extended abstracts on Human factors in computing systems. 2008, ACM: Florence, Italy.
26. Karapanos, E. and Martens, J.-B., The quantitative side of the Repertory Grid Technique: some concerns, in in the proceedings of the workshop Now Let's Do It in Practice: User Experience Evaluation Methods in Product Development, Human factors in computing systems, CHI'08. 2008: Florence.
27. Karapanos, E., Wensveen, S.A.G., Friederichs, B.H.W., and Martens, J.-B., Do knobs have character? Exploring diversity in users' inferences in CHI'08 extended abstracts on Human factors in computing systems. 2008, ACM Press: Florence.
28. Karapanos, E., Zimmerman, J., Forlizzi, J., and Martens, J.-B., User Experience Over Time: An initial framework, in Proceedings of the Twenty-Seventh Annual SIGCHI Conference on Human Factors in Computing Systems - CHI '09. 2009, ACM: Boston.
29. Markopoulos, P., Romero, N., van Baren, J., Ijsselsteijn, W., de Ruyter, B., and Farshchian, B. Keeping in touch with the family: home and away with the ASTRA awareness system. 2004: ACM Press New York, NY, USA.
30. Martens, J.-B., Image technology design: A perceptual approach. 2003, Boston: Kluwer Academic Publisher.
31. Osgood, C.E., Suci, G., and Tannenbaum, P., The measurement of meaning. 1957, Urbana, IL: University of Illinois Press.
32. Ramsay, J.O., MATFIT: A Fortran Subroutine for Comparing Two Matriced in a Subspace. Psychometrika, 1990. 55(3): p. 551--553.
33. Sarstedt, M., A review of recent approaches for capturing heterogeneity in partial least squares path modelling. Journal of Modelling in Management, 2008. 3(2): p. 140--161.
34. Schenkman, B.N. and Jönsson, F.U., Aesthetics and preferences of Web pages. Behaviour&Information Technology, 2000. 19: p. 367--377.
35. Steenkamp, J.-B.E.M. and Van Trijp, H.C.M., Attribute elicitation in Marketing Research: A comparison of Three Procedures. Marketing Letters, 1997. 8:2: p. 153--165.
36. van Kleef, E., van Trijp, H.C.M., and Luning, P., Consumer research in the early stages of new product development: a critical review of methods and techniques. Food Quality and Preference, 2005. 16(3): p. 181--201.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods