Semi-automatic video annotation based on active learning with multiple complementary predictors

ABSTRACT

In this paper, we will propose a novel semi-automatic annotation scheme for video semantic classification. It is well known that the large gap between high-level semantics and low-level features is difficult to be bridged by full-automatic content analysis mechanisms. To narrow down this gap, relevance feedback has been introduced in a number of literatures, especially in those works addressing the problem of image retrieval. And at the same time, active learning is also suggested to accelerate the converging speed of the learning process by labeling the most informative samples. Generally an active learning scheme includes a sample selection engine and a learning engine. In this paper, we will discuss the limitations of existing active learning algorithms and propose a novel active learning scheme based on multiple complementary predictors and incremental model adaptation, which improves the efficiencies of both of the primary components of active learning. Firstly, an efficient sample selection scheme is proposed, in which multiple predictors are applied to find most informative samples. Then an incremental model adaptation technique, maximum likelihood linear regression (MLLR), is used to update the classifiers which tackle the issue of unbalance between the original training set and the newly labeled data. It is proved that the samples selected by the proposed scheme are more representative than general active learning scheme, as well as the incremental model adaptation scheme is effective especially when the newly added data size is small.

REFERENCES

• 1: Greg Schohn , David Cohn, Less is More: Active Learning with Support Vector Machines, Proceedings of the Seventeenth International Conference on Machine Learning, p.839-846, June 29-July 02, 2000
• 2: David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
• 3: Chapelle. O, W.J., Scholkopf.B, Cluster kernels for semi-supervised learning. Advances in Neural Information Processing Systems, 2002.
• 4: Simon Tong , Daphne Koller, Support vector machine active learning with applications to text classification, The Journal of Machine Learning Research, 2, p.45-66, 3/1/2002
• 5: King-Shy Goh , Edward Y. Chang , Wei-Cheng Lai, Multimodal concept-dependent active learning for image retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027664]
• 6: Fang Qian, M.L., Wei-Ying Ma, Alternating Feature Spaces in Relevance Feedback. 1998.
• 7: Leggetter,C.J. and Woodland,P, Maximum likelihood linear regression for speaker adaptation of continuous density Hidden Markov Model. Computer Speech and Language.
• 8: Jun Wu , Xian-Sheng Hua , Hong-Jiang Zhang , Bo Zhang, An online-optimized incremental learning framework for video semantic classification, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027601]
• 9: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 10: Sally A. Goldman , Yan Zhou, Enhancing Supervised Learning with Unlabeled Data, Proceedings of the Seventeenth International Conference on Machine Learning, p.327-334, June 29-July 02, 2000
• 11: Minerva Yeung , Boon-Lock Yeo , Bede Liu, Segmentation of video by clustering and graph analysis, Computer Vision and Image Understanding, v.71 n.1, p.94-109, July 1, 1998 [doi>10.1006/cviu.1997.0628 ]
• 12: Yan Song, Xian-Sheng Hua, Li-Rong Dai, Ren-Hua Wang, Semi-Automatic Video Semantic Annotation Based on Active Learning. VCIP, 2005.
• 13: Guidelines for the TRECVID 2003 Evaluation http://www-nlpir.nist.gov/projects/tv2003/tv2003.htm
• 14: Yongmei Wang , Hongjiang Zhang, Content-Based Ima e Orientation Detection with Support Vector Machines, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'01), p.17, December 14-14, 2001
• 15: Ion Muslea , Steven Minton , Craig A. Knoblock, Selective Sampling with Co-Testing: Preliminary Results, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.1107, July 30-August 03, 2000
• 16: M.J.F.Gales, P.C.Woodland, "Mean and Variance Adaptation within the MLLR Framework", Computer Speech and Language Volume 10. 1996.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Experimentation

Keywords:
active learning, semantic classification, video annotation