skip to main content
10.1145/1168149.1168152acmotherconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Article

An explorative analysis of user evaluation studies in information visualisation

Published: 23 May 2006 Publication History

Abstract

This paper presents an analysis of user studies from a review of papers describing new visualisation applications and uses these to highlight various issues related to the evaluation of visualisations. We first consider some of the reasons why the process of evaluating visualisations is so difficult. We then dissect the problem by discussing the importance of recognising the nature of experimental design, datasets and participants as well as the statistical analysis of results. We propose explorative evaluation as a method of discovering new things about visualisation techniques, which may give us a better understanding of the mechanisms of visualisations. Finally we give some practical guidance on how to do evaluation correctly.

References

[1]
Bederson, B. B., Shneiderman, B., Wattenberg, M. Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies. ACM Transactions on Graphics, 21(4), Oct 2002, 833--854
[2]
Carroll, J. M, Rosson, M. B. Getting around the task-artifact cycle: how to make claims and design by scenario. ACM Transactions on Information Systems. Vol 10 No 2, April 1992, 181--212
[3]
Dix A., Brewster, S. Causing Trouble with Buttons. Ancilliary Proc. HCT'94, Glasgow, Scotland, 1994. http://www.hcibook.com/alan/papers/buttons94/
[4]
Dumais, S., Cutrell, E., Chen, H. Optimizing Search by Showing Results In Context. Proc. CHI'01, 2001, ACM Press, 277--284
[5]
Ellis, G. P., Bertini, E., Dix, A. The Sampling Lens:Making Sense of Saturated Visualisations, Proc. CHI'05 Extended Abstracts on Human Factors in Computing Systems, Portland, USA, 2005, ACM Press, 1351--1354
[6]
Fekete, J-D., Plaisant, C. Excentric Labeling: Dynamic Neighbourhood Labeling for Data Visualization. Proc. CHI'99, Pittsburgh, 1999, ACM Press, 512--519
[7]
Kosara, R., Healey, C. G., Interrante, V., Laidlaw, D. H., Ware, C. Thoughts on User Studies: Why, How, and When. Computer Graphics & Applications, 23(4), July 2003, 20--25
[8]
Lieberman, H. The Tyranny of Evaluation. (accessed 2006). http://web.media.mit.edu/~lieber/Misc/Tyranny-Evaluation.html
[9]
Mackinlay, J. D., Rao, R., Card, S. K. An Organic User Interface For Searching Citation Links, Proc. CHI'95, Denver, May 1995, ACM Press, 67--73
[10]
O'Donnell, R., Dix, A., Ball, L. Exploring The PieTree for Representing Numerical Hierarchical Data, Proc. HCI2006, London, Sept. 2006, Springer
[11]
Paek, T., Dumais, S., Logan, R. WaveLens: A New View onto Internet Search Results. Proc. CHI'04, Vienna, Austria, Apr 2004, ACM Press, 727--733
[12]
Pirolli, P., Schank, P., Hearst, M., Diehl, C. Scatter/Gather browsing communicates the topic structure of a very large text collection. Proc. CHI'96, Vancouver, May 1996, ACM Press, 213--220
[13]
Plaisant, C., Milash, B., Rose, A., Widoff, S., Shneiderman, B. LifeLines: Visualizing Personal Histories. Proc. CHI'96, 1996, ACM Press, 221--227
[14]
Plaisant, C. The Challenge of Information Visualization Evaluation. Advanced Visual interfaces, Italy, 2004, ACM Press
[15]
Tory, M., Möller, T. Evaluating Visualizations: Do Expert Reviews Work? IEEE Computer Graphics and Applications, 25(5), 2005, 8--11
[16]
Wong, N., Carpendale, S., Greenberg, S. EdgeLens: An Interactive Method for Managing Edge Congestion in Graphs. IEEE Symposium on Information Visualization, Oct 2003, 51--58
[17]
Yang, J., Ward, M. O., Rundensteiner, E. A., Huang, S. Interactive hierarchical displays: a general framework for visualization and exploration of large multivariate data sets. Computers and Graphics, 27(2), Apr 2003, 265--283
[18]
Zhai, S. Evaluation is the worst form of HCI research except all those other forms that have been tried. (accessed 2006). http://www.almaden.ibm.com/u/zhai/papers/EvaluationDemocracy.htm

Cited By

View all
  • (2024)CrimeSeen: An Interactive Visualization Environment for Scenario Testing on Criminal Cocaine NetworksComputational Science – ICCS 202410.1007/978-3-031-63759-9_24(195-204)Online publication date: 2-Jul-2024
  • (2023)Enhancing co-production of knowledge: Visualisation and engagement approaches for evidence-based decision making within the Kenya Agroforestry StrategyEnvironmental Science & Policy10.1016/j.envsci.2022.11.021140(152-162)Online publication date: Feb-2023
  • (2023)Evaluating narrative visualization: a survey of practitionersInternational Journal of Data Science and Analytics10.1007/s41060-023-00394-918:1(19-34)Online publication date: 31-Mar-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
BELIV '06: Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization
May 2006
89 pages
ISBN:1595935622
DOI:10.1145/1168149
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. case study
  2. evaluation
  3. explorative evaluation
  4. information visualisation

Qualifiers

  • Article

Conference

AVI06

Acceptance Rates

Overall Acceptance Rate 45 of 64 submissions, 70%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)110
  • Downloads (Last 6 weeks)9
Reflects downloads up to 09 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)CrimeSeen: An Interactive Visualization Environment for Scenario Testing on Criminal Cocaine NetworksComputational Science – ICCS 202410.1007/978-3-031-63759-9_24(195-204)Online publication date: 2-Jul-2024
  • (2023)Enhancing co-production of knowledge: Visualisation and engagement approaches for evidence-based decision making within the Kenya Agroforestry StrategyEnvironmental Science & Policy10.1016/j.envsci.2022.11.021140(152-162)Online publication date: Feb-2023
  • (2023)Evaluating narrative visualization: a survey of practitionersInternational Journal of Data Science and Analytics10.1007/s41060-023-00394-918:1(19-34)Online publication date: 31-Mar-2023
  • (2023)Visualization Psychology for Eye Tracking EvaluationVisualization Psychology10.1007/978-3-031-34738-2_10(243-260)Online publication date: 7-Nov-2023
  • (2022)Users’ Cognitive Load: A Key Aspect to Successfully Communicate Visual Climate InformationBulletin of the American Meteorological Society10.1175/BAMS-D-20-0166.1103:1(E1-E16)Online publication date: Jan-2022
  • (2022)PotteryVR: virtual reality potteryThe Visual Computer10.1007/s00371-022-02521-238:12(4035-4055)Online publication date: 20-Jun-2022
  • (2021)Covid-on-the-Web: Exploring the COVID-19 scientific literature through visualization of linked data from entity and argument miningQuantitative Science Studies10.1162/qss_a_001642:4(1301-1323)Online publication date: 1-Dec-2021
  • (2021)Experimental Evaluation of Virtual Pottery Systems2021 International Conference on Cyberworlds (CW)10.1109/CW52790.2021.00012(25-32)Online publication date: Sep-2021
  • (2021)EVALUATING PROTOTYPING SUPPORT IN EARLY TRANSFORMATIVE PSS DESIGNProceedings of the Design Society10.1017/pds.2021.1411(1411-1420)Online publication date: 27-Jul-2021
  • (2020)Towards Identification and Mitigation of Task-Based Challenges in Comparative Visualization Studies2020 IEEE Workshop on Evaluation and Beyond - Methodological Approaches to Visualization (BELIV)10.1109/BELIV51497.2020.00014(55-64)Online publication date: Oct-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media