Human performance measures for video retrieval

ABSTRACT

In this paper, we describe the challenges of assessing human performance during video retrieval episodes and describe several measures of human performance that have been used in developing visual surrogates for the Open Video Digital Library (http://www.open-video.org). These include two sets of cognitive performance measures that aim to assess human recognition and inference and a set of attitudinal measures that aim to assess user satisfaction with video surrogates.

REFERENCES

• 1: Boekelheide, K., Brown, A., Fu, X., Marchionini, G., Oh, S., Rogers, G., Saelim, B., Song, Y., & Stutzman, F. (2006). Audio surrogation for digital video: A design framework. SILS Technical Report TR-2006-02. http://sils.unc.edu/research/publications/reports/TR-2006-02.pdf
• 2: Peter Bruza , Robert McArthur , Simon Dennis, Interactive Internet search: keyword, directory and query reformulation mechanisms compared, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.280-287, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345598]
• 3: John P. Chin , Virginia A. Diehl , Kent L. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-218, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57203]
• 4: Christian Fluhr , Pierre-Alain Moéllic , Patrick Hede, Usage-oriented multimedia information retrieval technological evaluation, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178719]
• 5: Hughes, A., Wilkens, T., Wildemuth, B., & Marchionini, G. (2003). Text or pictures? An eyetracking study of how people view digital video surrogates. Proceedings of the International Conference on Image and Video Retrieval (CIVR 2003), 271--280.
• 6: Bernd Fröhlich , John Plate, The cubic mouse: a new device for three-dimensional input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.526-531, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332491]
• 7: Ying Li , Chitra Dorai , Robert Farrell, Creating MAGIC: system for generating learning object metadata for instructional content, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101227]
• 8: Gary Marchionini, Exploratory search: from finding to understanding, Communications of the ACM, v.49 n.4, April 2006 [doi>10.1145/1121949.1121979]
• 9: Gary Marchionini , Barbara M. Wildemuth , Gary Geisler, The Open Video Digital Library: A Möbius strip of research and practice, Journal of the American Society for Information Science and Technology, v.57 n.12, p.1629-1643, October 2006 [doi>10.1002/asi.v57:12]
• 10: Xiangming Mu , Gary Marchionini, Smartlinks in a video-based collaborative distance learning system: a cognitive model and evaluation study, University of North Carolina at Chapel Hill, Chapel Hill, NC, 2004
• 11: Over, P., Kraaij, W., & Smeaton, A. (2005). TRECVID 2005: An introduction. Proc. TRECVID 2005 (Gaithersburg, MD), 1-14. http://www.cdvp.dcu.ie/Papers/TRECVid2005_Overview.pdf
• 12: Reid, G.B. & Colle, H.A. (1988). Critical SWAT values for predicting operator overload. In Proceedings of the Human Factors Society 32nd annual meeting. Santa Monica, CA: Human Factors Society, 1414--1418.
• 13: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 14: Karen Sparck Jones, What's the value of TREC: is there a gap to jump or a chasm to bridge?, ACM SIGIR Forum, v.40 n.1, p.10-20, June 2006 [doi>10.1145/1147197.1147198]
• 15: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722 ]
• 16: Thijs Westerveld , Roelof van Zwol, Benchmarking multimedia search in structured collections, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178721]
• 17: White, R. & Marchionini, G. (in press). Examining the effectiveness of real-time query expansion. Information Processing & Management.
• 18: White, R., Muresan, G., & Marchionini, G. (in press). Evaluating exploratory search systems. SIGIR Workshop paper (Seattle, August 7-11, 2006).
• 19: Yang, M., Wildemuth, B. M., Marchionini, G., Wilkens, T., Geisler, G., Hughes, A., Gruss, R., & Webster, C. (2003b). Measures of User Performance in Video Retrieval Research. SILS Technical Report 2003-02. Chapel Hill, NC: University of North Carolina, School of Information and Library Science. http://www.ils.unc.edu/ils/research/TR-2003-02.pdf.
• 20: Yang, M., Wildemuth, B., Marchionini, G., Wilkens, T., Geisler, G., Hughes, A., Gruss, R., & Webster, C. (2003). Measuring user performance during interactions with digital video collections. Proc of the American Society for Information Science & Technology, (Long Beach, CA, Oct. 19-23, 2003), Silver Spring, MD: ASIST. 3-11.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

General Terms:
Experimentation, Human Factors, Measurement, Performance

Keywords:
exploratory search, human-computer information retrieval, information interaction, video retrieval