An explorative analysis of user evaluation studies in information visualisation

ABSTRACT

This paper presents an analysis of user studies from a review of papers describing new visualisation applications and uses these to highlight various issues related to the evaluation of visualisations. We first consider some of the reasons why the process of evaluating visualisations is so difficult. We then dissect the problem by discussing the importance of recognising the nature of experimental design, datasets and participants as well as the statistical analysis of results. We propose explorative evaluation as a method of discovering new things about visualisation techniques, which may give us a better understanding of the mechanisms of visualisations. Finally we give some practical guidance on how to do evaluation correctly.

References

1. Bederson, B. B., Shneiderman, B., Wattenberg, M. Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies. ACM Transactions on Graphics, 21(4), Oct 2002, 833--854
2. Carroll, J. M, Rosson, M. B. Getting around the task-artifact cycle: how to make claims and design by scenario. ACM Transactions on Information Systems. Vol 10 No 2, April 1992, 181--212
3. Dix A., Brewster, S. Causing Trouble with Buttons. Ancilliary Proc. HCT'94, Glasgow, Scotland, 1994. http://www.hcibook.com/alan/papers/buttons94/
4. Dumais, S., Cutrell, E., Chen, H. Optimizing Search by Showing Results In Context. Proc. CHI'01, 2001, ACM Press, 277--284
5. Ellis, G. P., Bertini, E., Dix, A. The Sampling Lens:Making Sense of Saturated Visualisations, Proc. CHI'05 Extended Abstracts on Human Factors in Computing Systems, Portland, USA, 2005, ACM Press, 1351--1354
6. Fekete, J-D., Plaisant, C. Excentric Labeling: Dynamic Neighbourhood Labeling for Data Visualization. Proc. CHI'99, Pittsburgh, 1999, ACM Press, 512--519
7. Kosara, R., Healey, C. G., Interrante, V., Laidlaw, D. H., Ware, C. Thoughts on User Studies: Why, How, and When. Computer Graphics & Applications, 23(4), July 2003, 20--25
8. Lieberman, H. The Tyranny of Evaluation. (accessed 2006). http://web.media.mit.edu/~lieber/Misc/Tyranny-Evaluation.html
9. Mackinlay, J. D., Rao, R., Card, S. K. An Organic User Interface For Searching Citation Links, Proc. CHI'95, Denver, May 1995, ACM Press, 67--73
10. O'Donnell, R., Dix, A., Ball, L. Exploring The PieTree for Representing Numerical Hierarchical Data, Proc. HCI2006, London, Sept. 2006, Springer
11. Paek, T., Dumais, S., Logan, R. WaveLens: A New View onto Internet Search Results. Proc. CHI'04, Vienna, Austria, Apr 2004, ACM Press, 727--733
12. Pirolli, P., Schank, P., Hearst, M., Diehl, C. Scatter/Gather browsing communicates the topic structure of a very large text collection. Proc. CHI'96, Vancouver, May 1996, ACM Press, 213--220
13. Plaisant, C., Milash, B., Rose, A., Widoff, S., Shneiderman, B. LifeLines: Visualizing Personal Histories. Proc. CHI'96, 1996, ACM Press, 221--227
14. Plaisant, C. The Challenge of Information Visualization Evaluation. Advanced Visual interfaces, Italy, 2004, ACM Press
15. Tory, M., Möller, T. Evaluating Visualizations: Do Expert Reviews Work? IEEE Computer Graphics and Applications, 25(5), 2005, 8--11
16. Wong, N., Carpendale, S., Greenberg, S. EdgeLens: An Interactive Method for Managing Edge Congestion in Graphs. IEEE Symposium on Information Visualization, Oct 2003, 51--58
17. Yang, J., Ward, M. O., Rundensteiner, E. A., Huang, S. Interactive hierarchical displays: a general framework for visualization and exploration of large multivariate data sets. Computers and Graphics, 27(2), Apr 2003, 265--283
18. Zhai, S. Evaluation is the worst form of HCI research except all those other forms that have been tried. (accessed 2006). http://www.almaden.ibm.com/u/zhai/papers/EvaluationDemocracy.htm

Index Terms

• Computing methodologies

  • Modeling and simulation

    • Simulation types and techniques

      • Visual analytics

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods