To search or to label?: predicting the performance of search-based automatic image classifiers

ABSTRACT

In this work we explore the trade-offs in acquiring training data for image classification models through automated web search as opposed to human annotation. Automated web search comes at no cost in human labor, but sometimes leads to decreased classification performance, while human annotations come at great expense in human labor but result in better performance. The primary contribution of this work is a system for predicting which visual concepts will show the greatest increase in performance from investing human effort in obtaining annotations. We propose to build this system as an estimation of the absolute gain in average precision (AP) experienced from using human annotations instead of web search. To estimate the AP gain, we rely on statistical classifiers built on top of a number of quality prediction features. We employ a feature selection algorithm to compare the quality of each of the predictors and find that cross-domain image similarity and cross-domain model generalization metrics are strong predictors, while concept frequency and within-domain model quality are weak predictors. In a test application, we find that the prediction scheme can result in a savings in annotation effort of up to 75\%, while only incurring marginal damage (10% relative decrease in mean average precision) to the overall performance of the concept models.

REFERENCES

• 1: NIST TREC Video Retrieval Evaluation http://www-nlpir.nist.gov/projects/trecvid/.
• 2: LSCOM lexicon definitions and annotations version 1.0, DTO challenge workshop on large scale concept ontology for multimedia. Technical report, Columbia University, March 2006.
• 3: A. Amir, J. Argillander, M. Campbell, A. Haubold, G. Iyengar, S. Ebadollahi, F. Kang, M. R. Naphade, A. Natsev, J. R. Smith, J. Tesic, and T. Volkmer. IBM Research TRECVID-2005 Video Retrieval System. In NIST TRECVID workshop, Gaithersburg, MD, November 2005.
• 4: C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/¿cjlin/libsvm.
• 5: S.-F. Chang, W. Hsu, L. Kennedy, L. Xie, A. Yanagawa, E. Zavesky, and D. Zhang. Columbia University TRECVID-2005 Video Search and High-Level Feature Extraction. In NIST TRECVID workshop, Gaithersburg, MD, November 2005.
• 6: R. Fergus , L. Fei-Fei , P. Perona , A. Zisserman, Learning Object Categories from Google"s Image Search, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1816-1823, October 17-20, 2005 [doi>10.1109/ICCV.2005.142]
• 7: G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4): 235--244, 1990.
• 8: C. G. Snoek, M. Worring, D. C.Koelma, and A. W. Smeulders. Learned lexicon-driven interactive video retrieval. In CIVR, 2006.
• 9: R. Typke, R. C. Veltkamp, and F. Wiering. A measure for evaluating retrieval techniques based on partially ordered ground truth lists. In ICME, 2006.
• 10: Timo Volkmer , John R. Smith , Apostol (Paul) Natsev, A web-based system for collaborative annotation of large image and video collections: an evaluation and user study, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101341]
• 11: Xin-Jing Wang , Lei Zhang , Feng Jing , Wei-Ying Ma, AnnoSearch: Image Auto-Annotation by Search, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1483-1490, June 17-22, 2006 [doi>10.1109/CVPR.2006.58]
• 12: Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076121]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation, Performance

Keywords:
performance prediction, search-based concept models