Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool

ABSTRACT

In this position paper we discuss successes and limitations of current evaluation strategies for scientific visualizations and argue for embracing a mixed methods strategy of evaluation. The most novel contribution of the approach that we advocate is a new emphasis on employing design processes as practiced in related fields (e.g., graphic design, illustration, architecture) as a formalized mode of evaluation for data visualizations. To motivate this position we describe a series of recent evaluations of scientific visualization interfaces and computer graphics strategies conducted within our research group. Complementing these more traditional evaluations our visualization research group also regularly employs sketching, critique, and other design methods that have been formalized over years of practice in design fields. Our experience has convinced us that these activities are invaluable, often providing much more detailed evaluative feedback about our visualization systems than that obtained via more traditional user studies and the like. We believe that if design-based evaluation methodologies (e.g., ideation, sketching, critique) can be taught and embraced within the visualization community then these may become one of the most effective future strategies for both formative and summative evaluations.

References

1. D. Acevedo, C. Jackson, F. Drury, and D. Laidlaw. Using visual design experts in critique-based evaluation of 2D vector visualization methods. IEEE Transactions on Visualization and Computer Graphics, 14(4):877--884, 2008.
2. B. Buxton. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2007.
3. S. Carpendale. Evaluating Information Visualizations. Information Visualization, pages 19--45. Springer-Verlag, Berlin, Heidelberg, 2008.
4. D. Coffey, N. Malbraaten, T. B. Le, I. Borazjani, F. Sotiropoulos, A. G. Erdman, and D. F. Keefe. Interactive Slice WIM: Navigating and interrogating volume data sets using a multisurface, multitouch VR interface. IEEE Transactions on Visualization and Computer Graphics, 18:1614--1626, 2012.
5. B. Jackson, D. Schroeder, and D. F. Keefe. Nailing down multi-touch: anchored above the surface interaction for 3D modeling and navigation. In Proceedings of the 2012 Graphics Interace Conference, GI '12, pages 181--184, Toronto, Ontario, Canada, 2012.
6. C. D. Jackson, D. Acevedo, D. H. Laidlaw, F. Drury, E. Vote, and D. Keefe. Designer-critiqued comparison of 2D vector visualization methods: a pilot study. In ACM SIGGRAPH 2003 Sketches & Applications, 2003.
7. D. F. Keefe, D. Acevedo, J. Miles, F. Drury, S. M. Swartz, and D. H. Laidlaw. Scientific sketching for collaborative VR visualization design. IEEE Transactions on Visualization and Computer Graphics, 14(4):835--847, 2008.
8. D. F. Keefe, D. B. Karelitz, E. L. Vote, and D. H. Laidlaw. Artistic collaboration in designing VR visualizations. IEEE Computer Graphics and Applications, 25(2):18--23, 2005.
9. S. Kim, H. Hagh-Shenas, and V. Interrante. Conveying three-dimensional shape with texture. In Proceedings of the 1st Symposium on Applied perception in graphics and visualization, APGV '04, pages 119--122, Los Angeles, California, 2004. ACM.
10. R. Kosara. Visualization criticism - the missing link between information visualization and art. In Proceedings of the 11th International Conference Information Visualization, IV '07, pages 631--636, Zurich, Switzerland, 2007. IEEE Computer Society.
11. R. Kosara, F. Drury, L. E. Holmquist, and D. H. Laidlaw. Visualization criticism. IEEE Computer Graphics and Applications, 28(3):13--15, 2008.
12. M. R. Mine, F. P. Brooks, Jr., and C. H. Sequin. Moving objects in space: exploiting proprioception in virtual-environment interaction. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH '97, pages 19--26, Los Angeles, CA, USA, 1997. ACM.
13. P. Saraiya, C. North, and K. Duca. An insight-based methodology for evaluating bioinformatics visualizations. IEEE Transactions on Visualization and Computer Graphics, 11(4):443--456, 2005.
14. D. A. Schon. Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. Jossey-Bass Higher Education Series. Wiley, Hoboken, NJ, USA.
15. J. Schulze, A. Forsberg, A. Kleppe, R. Zeleznik, and D. Laidlaw. Characterizing the effect of level of immersion on a 3D marking task. In Proceedings of HCI International, pages 447--452, Las Vegas, Nevada, USA, 2005.
16. L. Thorson, H. Sohn, J. Downing, A. Ellingson, D. Nuckley, and D. F. Keefe. A designers approach to scientific visualization: Visual strategies for illustrating motion datasets. In Poster Proceedings of IEEE VisWeek, Providence, RI, USA, 2011. IEEE Computer Society.
17. H. Woltring, R. Huiskes, A. de Lange, and F. Veldpaus. Finite centroid and helical axis estimation from noisy landmark measurements in the study of human joint kinematics. Journal of Biomechanics, 18(5):379--389, 1985.
18. L. Yu, P. Svetachov, P. Isenberg, M. H. Everts, and T. Isenberg. FI3D: Direct-touch interaction for the exploration of 3D scientific visualization spaces. IEEE Transactions on Visualization and Computer Graphics, 16(6):1613--1622, 2010.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods