Learning rich semantics from news video archives by style analysis

ABSTRACT

We propose a generic and robust framework for news video indexing which we founded on a broadcast news production model. We identify within this model four production phases, each providing useful metadata for annotation. In contrast to semiautomatic indexing approaches which exploit this information at production time, we adhere to an automatic data-driven approach. To that end, we analyze a digital news video using a separate set of multimodal detectors for each production phase. By combining the resulting production-derived features into a statistical classifier ensemble, the framework facilitates robust classification of several rich semantic concepts in news video; rich meaning that concepts share many similarities in their production process. Experiments on an archive of 120 hours of news video from the 2003 TRECVID benchmark show that a combined analysis of production phases yields the best results. In addition, we demonstrate that the accuracy of the proposed style analysis framework for classification of several rich semantic concepts is state-of-the-art.

REFERENCES

• 1: Adams, B., Dorai, C., and Venkatesh, S. 2002. Toward automatic extraction of expressive elements from motion pictures: Tempo. IEEE Trans. Multimedia 4, 4, 472--481.
• 2: Adams, W. H., Iyengar, G., Lin, C.-Y., Naphade, M., Neti, C., Nock, H., and Smith, J. 2003. Semantic indexing of multimedia content using visual, audio, and text cues. EURASIP J. Appl. Signal Process. 2003, 2, 170--185.
• 3: Amir, A., Berg, M., Chang, S.-F., Hsu, W., Iyengar, G., Lin, C.-Y., Naphade, M., Natsev, A., Neti, C., Nock, H., Smith, J., Tseng, B., Wu, Y., and Zhang, D. 2003. IBM research TRECVID-2003 video retrieval system. In Proceedings of the TRECVID Workshop. NIST Special Publication. Gaithersburg, Md.
• 4: Baan, J., Ballegooij, A., Geusebroek, J., Hiemstra, D., den Hartog, J., List, J., Snoek, C., Patras, I., Raaijmakers, S., Todoran, L., Vendrig, J., de Vries, A., Westerveld, T., and Worring, M. 2001. Lazy users and automatic video retrieval tools in (the) lowlands. In Proceedings of the 10th Text REtrieval Conference, E. Voorhees and D. Harman, eds. NIST Special Publication, vol. 500-250. Gaithersburg, Md.
• 5: Barry, B. and Davenport, G. 2003. Documenting life: Videography and common sense. In Proceedings of the IEEE International Conference on Multimedia & Expo. Baltimore, Md.
• 6: Boggs, J. and Petrie, D. 2000. The Art of Watching Films, 5th ed. Mayfield Publishing Mountain View, Calif.
• 7: Bordwell, D. and Thompson, K. 1997. Film Art: An Introduction, 5th ed. McGraw-Hill, New York.
• 8: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350 ]
• 9: Chang, C.-C. and Lin, C.-J. 2001. LIBSVM : A Library for Support Vector Machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
• 10: Marc Davis, Editing out Video Editing, IEEE MultiMedia, v.10 n.2, p.54-64, March 2003 [doi>10.1109/MMUL.2003.1195161]
• 11: Anind K. Dey, Understanding and Using Context, Personal and Ubiquitous Computing, v.5 n.1, p.4-7, February 2001 [doi>10.1007/s007790170019 ]
• 12: Paul Dourish, What we talk about when we talk about context, Personal and Ubiquitous Computing, v.8 n.1, p.19-30, February 2004 [doi>10.1007/s00779-003-0253-8]
• 13: Jean-Luc Gauvain , Lori Lamel , Gilles Adda, The LIMSI Broadcast News transcription system, Speech Communication, v.37 n.1-2, p.89-108, May 2002 [doi>10.1016/S0167-6393(01)00061-9 ]
• 14: Hanjalic, A., Lagendijk, R., and Biemond, J. 1999. Automated high-level movie segmentation for advanced video-retrieval systems. IEEE Trans. Circuits Syst. Video Technol. 9, 4, 580--588.
• 15: Hauptmann, A., Baron, R., Chen, M.-Y., Christel, M., Duygulu, P., Huang, C., Jin, R., Lin, W.-H., Ng, T., Moraveji, N., Papernick, N., Snoek, C., Tzanetakis, G., Yang, J., Yan, R., and Wactlar, H. 2003. Informedia at TRECVID 2003: Analyzing and searching broadcast news video. In Proceedings of the TRECVID Workshop. NIST Special Publication. Gaithersburg, Md.
• 16: Anil K. Jain , Robert P. W. Duin , Jianchang Mao, Statistical Pattern Recognition: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.1, p.4-37, January 2000 [doi>10.1109/34.824819 ]
• 17: Frank Nack , Wolfgang Putz, Saying What it Means: Semi-Automated (News) Media Annotation, Multimedia Tools and Applications, v.22 n.3, p.263-302, March 2004 [doi>10.1023/B:MTAP.0000017031.26875.f7]
• 18: Naphade, M., Kozintsev, I., and Huang, T. 2002. A factor graph framework for semantic video indexing. IEEE Trans. Circuits Syst. Video Technol. 12, 1, 40--52.
• 19: Platt, J. 2000. Probabilities for SV machines. In Advances in Large Margin Classifiers, A. Smola, et al., eds. MIT Press, Cambridge, Mass., 61--74.
• 20: Quénot, G., Moraru, D., Besacier, L., and Mulhem, P. 2002. CLIPS at TREC -11: Experiments in video retrieval. In Proceedings of the 11th Text REtrieval Conference, E. Voorhees and L. Buckland, eds. NIST Special Publication, vol. 500-251. Gaithersburg, Md.
• 21: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 22: Toshio Sato , Takeo Kanade , Ellen K. Hughes , Michael A. Smith , Shin'ichi Satoh, Video OCR: indexing digital new libraries by recognition of superimposed captions, Multimedia Systems, v.7 n.5, p.385-395, September 1999 [doi>10.1007/s005300050140]
• 23: Robert E. Schapire, The Strength of Weak Learnability, Machine Learning, v.5 n.2, p.197-227, Jun. 1990 [doi>10.1023/A:1022648800760]
• 24: Henry Schneiderman , Takeo Kanade, Object Detection Using the Statistics of Parts, International Journal of Computer Vision, v.56 n.3, p.151-177, February-March 2004 [doi>10.1023/B:VISI.0000011202.85607.00]
• 25: Smeaton, A., Kraaij, W., and Over, P. 2003. TRECVID 2003---An introduction. In Proceedings of the TRECVID Workshop. NIST Special Publication. Gaithersburg, Md.
• 26: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972 ]
• 27: J. R. Smith , C. -S. Li, Decoding Image Semantics Using Composite Region Templates, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.9, June 21-21, 1998
• 28: Snoek, C. and Worring, M. 2005a. Multimedia event-based video indexing using time intervals. IEEE Trans. Multimedia 7, 4, 638--647.
• 29: Cees G. M. Snoek , Marcel Worring, Multimodal Video Indexing: A Review of the State-of-the-art, Multimedia Tools and Applications, v.25 n.1, p.5-35, January 2005 [doi>10.1023/B:MTAP.0000046380.27575.a5]
• 30: Snoek, C., Worring, M., and Hauptmann, A. 2004. Detection of TV news monologues by style analysis. In Proceedings of the IEEE International Conference on Multimedia & Expo. Taipei, Taiwan.
• 31: Sundaram, H. and Chang, S.-F. 2002. Computable scenes and structures in films. IEEE Trans. Multimedia 4, 4, 482--491.
• 32: Ba Tu Truong , Svetha Venkatesh , Chitra Dorai, Extraction of Film Takes for Cinematic Analysis, Multimedia Tools and Applications, v.26 n.3, p.277-298, August 2005 [doi>10.1007/s11042-005-0892-z]
• 33: Tseng, B., Lin, C.-Y., Naphade, M., Natsev, A., and Smith, J. 2003. Normalized classifier fusion for semantic visual concept detection. In Proceedings of the IEEE International Conference on Image Processing. vol. 2. Barcelona, Spain, 535--538.
• 34: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 35: Vendrig, J. and Worring, M. 2002. Systematic evaluation of logical story unit segmentation. IEEE Trans. Multimedia 4, 4, 492--499.
• 36: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722 ]
• 37: David H. Wolpert, Stacked generalization, Neural Networks, v.5 n.2, p.241-259, 1992 [doi>10.1016/S0893-6080(05)80023-1]
• 38: Lexing Xie , Peng Xu , Shih-Fu Chang , Ajay Divakaran , Huifang Sun, Structure analysis of soccer video with domain knowledge and hidden Markov models, Pattern Recognition Letters, v.25 n.7, p.767-775, May 2004 [doi>10.1016/j.patrec.2004.01.005]
• 39: Zhang, H.-J., Wu, J., Zhong, D., and Smoliar, S. 1997. An integrated system for content-based video retrieval and browsing. Pattern Recogn. 30, 4, 643--658.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning

General Terms:
Algorithms, Management, Performance

Keywords:
Benchmark evaluation, multimedia understanding, multimodal detectors, news video indexing, semantic classification, statistical pattern recognition, style analysis