skip to main content
research-article

The evaluation of ARCO: a lesson in curatorial competence and intuition with new technology

Published:01 July 2008Publication History
Skip Abstract Section

Abstract

This article presents the results of the evaluation study of the Augmented Representation of Cultural Objects (ARCO) system which provides software and interface tools to museum curators for the development of virtual museum exhibitions for the World Wide Web or for information kiosks. The aim of the current research is to investigate how a virtual museum system is coping with the demands of museum curators, to examine the needs of virtual museum visitors, and to provide a set of criteria and guidelines for defining effective evaluation of such systems. Evaluation methods such as heuristic evaluation and walkthroughs were employed in the study in order to assess various components and interfaces of the system.

References

  1. ARCO Evaluation Report. 2004. Assessment and Evaluation report on the ARCO system and its components. Retrieved July, 1, 2007: http://www.arco-web.org/TextVersion/Documents/Deliverables/Arco-D16-R-1.0-170904.pdfGoogle ScholarGoogle Scholar
  2. Arnold, D. and G. Geser. 2007. Research Agenda for the Applications of ICT to Cultural Heritage. Epoch Publication. Budapest: Archaeolingua.Google ScholarGoogle Scholar
  3. Billinghurst, M., Kato, H., and I. Poupyrev. 2001. The Magic-Book---Moving Seamlessly between Reality and Virtuality. IEEE Computer Graphics and Applications 21 (3): 2--4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Borun, M. and R. Korn, eds. 1999. Introduction to Museum Evaluation. Committee on Audience Research and Evaluation, American Association of Museums, Professional Practice Series, Washington, DC: AAM.Google ScholarGoogle Scholar
  5. Catarci, T. 2000. What Happened when Database Researchers Met Usability. Information Systems, 25 (3): 177--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. CIDOC Multimedia Working Group. 1997. Multimedia Evaluation Criteria. (Chair: J. Trant). Retrieved July, I, 2007: http://www.archimuse.com/papers/cidoc/cidoc.mmwg.eval.crit.htmlGoogle ScholarGoogle Scholar
  7. Di Bias, N., Guermand, M. P., Orsini, C., and P. Paolini. 2002. Evaluating the Features of Museum Websites (The Bologna Report). Retrieved July, 1, 2007: http://www.archimuse.com/mw2002/papers/diblas/diblas.htmlGoogle ScholarGoogle Scholar
  8. Di Blas, N., Garzotto, F., Guermand, M. P., and F. Nicolucci. 2003. A Methodology for Evaluating Archaeological Web Sites. In K. Fischer Ausserer, W. Börner, M. Goriany and L. Karlhuber-Vöckl (Eds.), Proceedings of CAA (Computer Applications and Quantitative Methods in Archaeology) (pp. 200--203). BAR International: British Archaeological Series.Google ScholarGoogle Scholar
  9. Dumas, J. F., and J. C. Redish. 1999. A practical guide to usability testing. Portland, Intellect Ltd. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Economou, M. 2002. XII: Assessment of Projects by User Evaluation. In Guide to Good Practice in Digital Representation and Management of Cultural Heritage Material' by the National Initiative for a Networked Cultural Heritage, USA. Retrieved January 22, 2006 on the World Wide Web: http://www.nvu.edu/its/humanities/ninchguide/Google ScholarGoogle Scholar
  11. Greenbaum, J., and M. Kyng, eds. 1991. Design at Work: Cooperative Design of Computer Systems. Hillsdale, NJ: Lawrence Erlbaum. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Guest G., Bunce A. and L. Johnson. 2006. How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods 18: 59.Google ScholarGoogle ScholarCross RefCross Ref
  13. Hein, G. E. 1998. Learning in the museum. London: Routledge.Google ScholarGoogle Scholar
  14. Hooper-Greenhill, E. 2000. Changing Values in the Art Museum: Rethinking Communication and Learning. International Journal of Heritage Studies 6(1): 9--31.Google ScholarGoogle ScholarCross RefCross Ref
  15. Hoptman, G. H. 1992. The Virtual Museum and Related Epistemological Concerns. Sociomedia. Multimedia, Hypermedia and the Social Construction of Knowledge: 141--159.Google ScholarGoogle Scholar
  16. IBM a, User-Centered Design principles. Retrieved July, 1, 2007: http://www-03.ibm.com/easy/page/13Google ScholarGoogle Scholar
  17. IBM b, Design basics. Retrieved July, 1, 2007: http://www-03.ibm.com/easy/page/6Google ScholarGoogle Scholar
  18. ICOM/CIDOC. Multimedia Working Group. 1996. Introduction to Multimedia in Museums. Retrieved July, 1, 2007: http://www.willpowerinfo.myby.co.uk/cidoc/introtomultimediamuseums.pdfGoogle ScholarGoogle Scholar
  19. ISO (International Organization for Standardization) 1998. ISO 9241: Ergonomic requirements for office work with visual display terminals (VDTs).Google ScholarGoogle Scholar
  20. Jones, J., and M. Christal. 2002. The Future of Virtual Museums: On-Line, Immersive, 3D Environments. Created Realities Group white paper. Retrieved July, 1, 2007: http://created-realities.com/pdf/Virtual_Museums.pdfGoogle ScholarGoogle Scholar
  21. Karasimos, A. and Isard, A. 2004 'Multilingual evaluation for a natural language generation system;, Proceedings of Lanugage Resources and Evaluation (LREC) 2004Google ScholarGoogle Scholar
  22. Karat, C., Campbell, R. & Fiegel, T. 1992. Comparison of Empirical Testing and Walkthrough Methods in User Interface Evaluation. In Proceedings of ACM CHI '92. Monterey, CA, May 3--7, 397--404, ACM publ. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Karoulis A., Sylaiou S., White M. 2006a. Combinatory Usability Evaluation of an Educational Virtual Museum Interface. In Proceedings of the 6th IEEE International Conference on Advanced Learning Technologies (ICALT2006), Kerkrade, The Netherlands, July 2006: 340--342. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Karoulis, A., Sylaiou, S., and M. White, 2006b. Usability evaluation of a virtual museum interface. Journal Informatica. 17(3): 363--380. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Liarokapis, F., Sylaiou, S., Basu, A., Mourkoussis, N., White, M., Lister, P. F. 2004. An Interactive Visualisation Interface for Virtual Museums. In Proceedings of the 5th International Symposium on Virtual Reality, Archaeology and Cultural Heritage, Brussels, Belgium, December 2004: 47--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Lewis, C., Polson P., Wharton C., and J. Rieman. 1990. Testing walkthrough methodology for theory-based design of walkup-and-use interfaces. In Chew, J. and Whiteside, J. (Eds.), Proceedings ACM CHI '90 Conference Seattle (Special issue of the SIGCHI Bulletin): 235--242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Liarokapis, F., Sylaiou, S., Basu, A., Mourkoussis, N., White, M., and P. F. Lister. 2004. An Interactive Visualisation Interface for Virtual Museums. In Fellner, D. and Spencer S. (Eds), Proceedings of the 5th International Symposium on Virtual Reality, Archaeology and Cultural Heritage: 47--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Lord, G. D., Lord, B. and J. S. Nicks. 1989a. The Cost of Collecting. Museums Journal, 89(2): 32--33.Google ScholarGoogle Scholar
  29. Lord, G. D., Lord, B. and J. S. Nicks. 1989b. The Cost of Collecting. Collection Management in UK Museums - A Report commissioned by the Office of Arts and Libraries. London: HMSO.Google ScholarGoogle Scholar
  30. Mazzoleni, P., Bertino E., Ferrari E. and S. Valtolina. 2004. CiVeDi: Customized Virtual Environment for Database Interaction. ACM Sigmod Record, 33(3): 15--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Monopoli, M and Nicholas, D. 2001. 'A user evaluation of subject based information gateways: A case study ADAM' Aslib Proceedings Vol 53 (1) January 2001: 39--52.Google ScholarGoogle Scholar
  32. Mourkoussis, N., White M., Patel M., Chmielewski J. and K. Walczak. 2003. AMS - Metadata for Cultural Exhibitions using Virtual Reality. In Sutton, S., Greenberg, J., and Tennis, J. (Eds.), Proceedings of Dublin Core Conference (DC2003). Seattle, Washington, USA 2003. Retrieved July, 1, 2007: http://purl.oclc.org/dc2003/03mourkoussis.pdf Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Namioka, A. and D. Schuler, eds. 1993. Participatory design. Principles and practices. Hillsdale NJ: Lawrence Erlbaum. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Nicks, J. 1999. Collections Management. In Lord, G. D. and Lord, B. (Eds) The Manual of Museum Planning. London: The Stationery Office, 2nd edition: 109--139.Google ScholarGoogle Scholar
  35. Nielsen, J. 1993. Usability Engineering, New York, NY: Academic Press Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Norman, D. A. and S. W. Draper, eds. 1986. User Centered System Design. Hillsdale, NJ: Lawrence Erlbaum.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Pearce, S. M. 1992. Museums, Objects and Collections: A Cultural Study. Leicester and London: Leicester University Press.Google ScholarGoogle Scholar
  38. Preece, J., Roger, Y., Sharp, H., Beyon, D., Holland, S., and Carey, I., (1994). Human Computer Interaction. Wokingham, UK: Addison Wesley. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Preece, J., Rogers, Y., and H. Sharp. 2002. Interaction Design: Beyond Human-Computer Interaction. New York: John Wiley and Sons Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Romney, A. K., Weller, S. C and W. H. Batchelder. 1986. Culture as consensus: A theory of culture and informant accuracy. American Anthropologist, 88: 313--338.Google ScholarGoogle ScholarCross RefCross Ref
  41. Ryan, G., and H. R. Bernard. 2003. Techniques to identify themes. Field Methods 15: 85--109.Google ScholarGoogle ScholarCross RefCross Ref
  42. Scali, G., Segbert M. and B. Morganti. 2002. Multimedia applications for innovation in cultural heritage: 25 European trial projects and their accompanying measure TRIS. In 68th IFLA Council and General Conference August 18--24, 2002. Retrieved July, 1, 2007: http://www.ifla.org/IV/ifla68/papers/044--123e.pdfGoogle ScholarGoogle Scholar
  43. Signore, O. 2005. A Comprehensive Model for Web Sites Quality. In Proceedings of the Seventh IEEE International Symposium on Web Site Evolution, Budapest, Hungary: 30--36. Retrieved July, 1, 2007: http://www.weblab.isti.cnr.it/papers/restricted/wse2005.pdf) Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Sylaiou S., Almosawi A., Mania K., White M. 2004. Preliminary Evaluation of the Augmented Representation of Cultural Objects System. In Proceedings of the 10th International Conference on Virtual Systems and Multimedia, Hybrid Realities-Digital Partners, Explorations in Art, Heritage, Science and the Human Factor (VSMM04), November 2004, Softopia, Ogaki City, Japan: 426--431.Google ScholarGoogle Scholar
  45. Treinen, H. 1996. Ausstellungen und Kommunikationstheorie. In Haus der Geschichte der Bundesrepublik Deutschland (Hrsg.), Museen und ihre Besucher, Berlin: Argon.Google ScholarGoogle Scholar
  46. Tsapatori M. 2003. ORION Research Roadmap, Evaluation and Assessment, Object Rich Information Network (ORION), Deliverable 8. Retrieved July, 1, 2007: http://www.orionnet.org/Admin/LibraryLoc/file28.pdfGoogle ScholarGoogle Scholar
  47. Vergo, P. (ed.) 1989. The New Museology. London: Reaktion Books.Google ScholarGoogle Scholar
  48. Virtual Showcases. 2004. Evaluation Report. Retrieved July, 1, 2007: http://www.virtualshowcases.com/Technical/Deliverables.aspGoogle ScholarGoogle Scholar
  49. Wang, Y., L. M. Aroyo, N. Stash, and L. Rutledge, 2007. 'Interactive User modelling for personalized access to museum collections: The Rijksmuseum case study' in C. Conati, K. McCoy, and G. Paliouras (Eds) User Modeling 2007, Vol. 4511/2007, Berlin / Heidelberg: Springer-Verlag: 385--389. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. White M., Mourkoussis, N., Darcy, J., Petridis, P., Liarokapis, F., Lister, P. F., Walczak, K., Wojciechowski, R., Cellary, W., Chmielewski, J., Stawniak, M., Wiza, W., Patel, M., Stevenson, J., Manley, J., Giorgini, F., Sayd, P., and F. Gaspard. 2004. ARCO---An Architecture for Digitization, Management and Presentation of Virtual Exhibitions. In Proceedings of the 22nd International Conference on Computer Graphics (CGI '2004), Hersonissos, Crete, June 2004, Los Alamitos, California: IEEE Computer Society: 622--625. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Wojciechowski, R., Walczak, K., White, M. and W. Cellary. 2004. Building Virtual and Augmented Reality Museum Exhibitions. In Spencer S. N. (ed.), Proceedings of the Web3D 2004 Symposium -the 9th International Conference on 3D Web Technology, ACM SIGGRAPH, Monterey, California (USA): 135--144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Wolf, R. L. 1980. A naturalistic view of evaluation. Museum News, 58(1): 39--45.Google ScholarGoogle Scholar

Index Terms

  1. The evaluation of ARCO: a lesson in curatorial competence and intuition with new technology

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image Computers in Entertainment
      Computers in Entertainment   Volume 6, Issue 2
      Theoretical and Practical Computer Applications in Entertainment
      April/June 2008
      225 pages
      EISSN:1544-3574
      DOI:10.1145/1371216
      Issue’s Table of Contents

      Copyright © 2008 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 1 July 2008
      • Accepted: 1 April 2008
      • Revised: 1 February 2008
      • Received: 1 November 2007

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Popular
      • Editor picked

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format