Evaluation strategies for image understanding and retrieval

ABSTRACT

We address evaluation of image understanding and retrieval large scale image data in the context of three evaluation projects. The first project is a comprehensive strategy for evaluating image retrieval algorithms and provides an open reference data set for doing so. The second project develops word prediction as a semantically relevant evaluation strategy, and applies it to the evaluation of of image processing methods for semantic image analysis. The third project evaluates words for suitability of their visual properties for use in an image annotation framework.

REFERENCES

• 1: The benchathlon network, http://www.benchathlon.net.
• 2: The gnu image-finding tool, www.gnu.org/software/gift.
• 3: L. H. Armitage and P. G. B. Enser. Analysis of user need in image archives. Journal of Information Science, 23(4):287--299, 1997.
• 4: K. Barnard, P. Duygulu, and D. Forsyth. Clustering art. In IEEE Conference on Computer Vision and Pattern Recognition, pages II:434--441, 2001.
• 5: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 6: K. Barnard, P. Duygulu, R. Guru, P. Gabbur, and D. Forsyth. The effects of segmentation and feature choice in a translation model of object recognition. In Proc. of IEEE Computer Vision and Pattern Recognition, pages II:675--682, 2003.
• 7: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In International Conference on Computer Vision, pages II:408--415, 2001.
• 8: Sudhir Borra , Sudeep Sarkar, A Framework for Performance Characterization of Intermediate-Level Grouping Modules, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.11, p.1306-1312, November 1997 [doi>10.1109/34.632991 ]
• 9: P. Carbonetto, N. d. Freitas, and K. Barnard. A statistical model for general contextual object recognition. In European Conference on Computer Vision, 2004.
• 10: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800 ]
• 11: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800 ]
• 12: Thomas F. Coleman , Yuying Li, On the convergence of interior-reflective Newton methods for nonlinear minimization subject to bounds, Mathematical Programming: Series A and B, v.67 n.2-17, p.189-224, Nov. 1994
• 13: Thomas F. Coleman , Yuying Li, A Reflective Newton Method for Minimizing a Quadratic Function Subject to Bounds on some of the Variables, SIAM Journal on Optimization, v.6 n.4, p.1040-1058, 1996 [doi>10.1137/S1052623494240456]
• 14: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236 ]
• 15: I. J. Cox, M. L. Miller, T. P. Minka, T. V. Papathomas, and P. N. Yianilos. The Bayesian image retrieval system, PicHunter: Theory, implementation and psychophysical experiments. IEEE Transactions on Image Processing, 9(1):20--35, 2000.
• 16: Yining Deng , b. s. Manjunath, Unsupervised Segmentation of Color-Texture Regions in Images and Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.8, p.800-810, August 2001 [doi>10.1109/34.946985 ]
• 17: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 18: P. G. B. Enser. Query analysis in a visual information retrieval context. Journal of Document and Text Management, 1(1):25--39, 1993.
• 19: P. G. B. Enser. Progress in documentation pictorial information retrieval. Journal of Documentation, 51(2):126--170, 1995.
• 20: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146 ]
• 21: D. A. Forsyth. Benchmarks for storage and retrieval in multimedia databases. In Storage and Retrieval for Media Databases III, volume 4676. SPIE, 2002.
• 22: Venkat N. Gudivada , Vijay V. Raghavan, Content-Based Image Retrieval Systems, Computer, v.28 n.9, p.18-22, September 1995 [doi>10.1109/2.410145 ]
• 23: N. J. Gunther and G. B. Beratta. Benchmark for image retrieval using distributed systems over the internet: Birds-i. In G. B. Beretta and R. Schettini, editors, Internet Imaging III, volume 4311, pages 252--267. SPIE, 2001.
• 24: N. A. Heard and A. Smith. Bayesian piecewise polynomial modeling of ogive and unimodal curves. Technical report, 2002.
• 25: N. A. Heard and A. Smith. Bayesian piecewise polynomial modeling of ogive and unimodal curves. Technical report, 2002.
• 26: C. Holmes and N. Heard. Generalized monotonic regression using random change points. Statistics in Medicine, 22(4):623--638, 2003.
• 27: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 28: J. Jeon and R. Manmatha. Using maximum entropy for automatic image annotation. In In Proc. International Conference on Image and Video Retrieval (CIVR-2004), pages 24--32, 2004.
• 29: M. La Cascia , S. Sethi , S. Sclaroff, Combining Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.24, June 21-21, 1998
• 30: V. Lavrenko, R. Manmatha, and J. Jeon. A model for learning the semantics of pictures. In In the Proceedings of NIPS03, 2003.
• 31: Wei-Ying Ma , B. S. Manjunath, NeTra: a toolbox for navigating large image databases, Multimedia Systems, v.7 n.3, p.184-198, May 1999 [doi>10.1007/s005300050121 ]
• 32: Marjo Markkula , Eero Sormunen, End-User Searching Challenges Indexing Practices inthe Digital Newspaper Photo Archive, Information Retrieval, v.1 n.4, p.259-285, January 2000 [doi>10.1023/A:1009995816485 ]
• 33: David R. Martin , Charless C. Fowlkes , Jitendra Malik, Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.5, p.530-549, May 2004 [doi>10.1109/TPAMI.2004.1273918]
• 34: D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International Conference on Computer Vision, 2001.
• 35: N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller. Equations of state calculations by fast computing machines. Journal of Chemical Physics, 21:1087--1091, 1953.
• 36: A. Pentland , R. W. Picard , S. Sclaroff, Photobook: content-based manipulation of image databases, International Journal of Computer Vision, v.18 n.3, p.233-254, June 1996 [doi>10.1007/BF00123143]
• 37: T. Pfund and S. Marchand-Maillet. Dynamic multimedia annotation tool. In G. B. Beretta and R. Schettini, editors, Internet Imaging III, volume 4672, pages 206--224. SPIE, 2002.
• 38: T. Robertson, F. Wright, and R. Dykstra. Order-Restricted statistical Inference. Wiley: New York, 1998.
• 39: Gerard Salton, The state of retrieval system evaluation, Information Processing and Management: an International Journal, v.28 n.4, p.441-449, July-Aug. 1992 [doi>10.1016/0306-4573(92)90002-H ]
• 40: M. Schell and B. Singh. The reduced monotonic regression method. Journal of the American Statistical Association, 92(437):128--135, 1997.
• 41: Stan Sclaroff , Leonid Taycher , Marco La Cascia, ImageRover: A Content-Based Image Browser for the World Wide Web, Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '97), p.2, June 20-20, 1997
• 42: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688 ]
• 43: Nikhil V. Shirahatti , Kobus Barnard, Evaluating Image Retrieval, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.955-961, June 20-26, 2005 [doi>10.1109/CVPR.2005.147]
• 44: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972 ]
• 45: J. R. Smith, Image Retrieval Evaluation, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.112, June 21-21, 1998
• 46: Julia Vogel , Bernt Schiele, On Performance Characterization and Optimization for Image Retrieval, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.49-66, May 28-31, 2002
• 47: James Z. Wang , Jia Li, Learning-based linguistic indexing of pictures with 2--d MHMMs, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641104]
• 48: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109 ]
• 49: Keiji Yanai , Kobus Barnard, Image region entropy: a measure of "visualness" of web images associated with one concept, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101241]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Algorithms, Design, Experimentation, Human Factors

Keywords:
evaluation, image recognition, image retrieval