Player action recognition in broadcast tennis video with applications to semantic analysis of sports game

ABSTRACT

Recognition of player actions in broadcast sports video is a challenging task due to low resolution of the players in video frames. In this paper, we present a novel method to recognize the basic player actions in broadcast tennis video. Different from the existing appearance-based approaches, our method is based on motion analysis and considers the relationship between the movements of different body parts and the regions in the image plane. A novel motion descriptor is proposed and supervised learning is employed to train the action classifier. We also propose a novel framework by combining the player action recognition with other multimodal features for semantic and tactic analysis of the broadcast tennis video. Incorporating action recognition into the framework not only improves the semantic indexing and retrieval performance of the video content, but also conducts highlights ranking and tactics analysis in tennis matches, which is the first solution to our knowledge for tennis game. The experimental results demonstrate that our player action recognition method outperforms existing appearance-based approaches and the multimodal framework is effective for broadcast tennis video analysis.

REFERENCES

• 1: Chua Hock Chuan, Automatic Parsing of TV Soccer Programs, Proceedings of the International Conference on Multimedia Computing and Systems, p.167, May 15-18, 1995
• 2: A. Ekin, A.M. Tekalp, R. Mehrotra. Automatic soccer video analysis and summarization. IEEE Transaction on Image Processing, vol. 12, no. 7, pp. 796--807, 2003.
• 3: Lexing Xie , Peng Xu , Shih-Fu Chang , Ajay Divakaran , Huifang Sun, Structure analysis of soccer video with domain knowledge and hidden Markov models, Pattern Recognition Letters, v.25 n.7, p.767-775, May 2004 [doi>10.1016/j.patrec.2004.01.005]
• 4: Jürgen Assfalg , Marco Bertini , Carlo Colombo , Alberto Del Bimbo , Walter Nunziati, Semantic annotation of soccer videos: automatic highlights identification, Computer Vision and Image Understanding, v.92 n.2-3, p.285-305, November/December 2003 [doi>10.1016/j.cviu.2003.06.004]
• 5: N. Babaguchi, Y. Kawai, T. Ogura, T. Kitahashi. Personalized abstraction of broadcasted American football video by highlight selection. IEEE Transaction on Multimedia, vol. 6, no. 4, pp. 575--586, 2004.
• 6: K. Wan, J. Wang, C. Xu, Q. Tian. Automatic sports highlights extraction with content augmentation. Pacific-Rim Conference on Multimedia, vol. 3332, pp. 19--26, 2004.
• 7: G. S. Pingali, Y. Jean, A. Opalach, I. Carlbom. LucentVision: converting real world events into multimedia experiences. IEEE International Conference on Multimedia and Expo, vol. 3, pp. 1433--1436, 2000.
• 8: Ramesh Jain , Mubarak Shah, Motion-Based Recognition, Kluwer Academic Publishers, Norwell, MA, 1997
• 9: D. M. Gavrila, The visual analysis of human movement: a survey, Computer Vision and Image Understanding, v.73 n.1, p.82-98, Jan. 1999 [doi>10.1006/cviu.1998.0716 ]
• 10: Ross Cutler , Larry S. Davis, Robust Real-Time Periodic Motion Detection, Analysis, and Applications, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.781-796, August 2000 [doi>10.1109/34.868681 ]
• 11: Alexei A. Efros , Alexander C. Berg , Greg Mori , Jitendra Malik, Recognizing Action at a Distance, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.726, October 13-16, 2003
• 12: Aaron F. Bobick , James W. Davis, The Recognition of Human Movement Using Temporal Templates, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.3, p.257-267, March 2001 [doi>10.1109/34.910878 ]
• 13: Yang Song , Luis Goncalves , Pietro Perona, Unsupervised Learning of Human Motion, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.7, p.814-827, July 2003 [doi>10.1109/TPAMI.2003.1206511]
• 14: Hisashi Miyamori , Shu-ichi Iisaku, Video Annotation for Content-based Retrieval using Human Behavior Analysis and Domain Knowledge, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.320, March 26-30, 2000
• 15: H. Miyamori. Improving accuracy in behavior identification for content-based retrieval by using audio and video information. IEEE International Conference on Pattern Recognition, vol. 2, pp. 826--830, 2002.
• 16: G. Sudhir , John C. M. Lee , Anil K. Jain, Automatic Classification of Tennis Video for High-level Content-based Retrieval, Proceedings of the 1998 International Workshop on Content-Based Access of Image and Video Databases (CAIVD '98), p.81, January 03-03, 1998
• 17: G. S. Pingali , Y. Jean , I. Carlbom, Real-Time Tracking for Enhanced Tennis Broadcasts, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.260, June 23-25, 1998
• 18: X. Yu, C.H. Sim, J.R. Wang, L.F. Cheong. A trajectory-based ball detection and tracking algorithm in broadcast tennis video. IEEE International Conference on Image Processing, vol. 2, pp. 1049--1052, 2004.
• 19: E. Kijak , G. Gravier , P. Gros , L. Oisel , F. Bimbot, HMM based structuring of tennis videos using visual and audio cues, Proceedings of the 2003 International Conference on Multimedia and Expo - Volume 3 (ICME '03), p.309-312, July 06-09, 2003
• 20: M. Xu, L.Y. Duan, C.S. Xu, Q. Tian. A fusion scheme of visual and auditory modalities for event detection in sports video. IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3, pp. 189--192, 2003.
• 21: M. Petkovic, V. Mihajlovic, W. Jonker. Techniques for automatic video content derivation. IEEE International Conference on Image Processing, vol. 2, pp. 611--614, 2003.
• 22: P. Wang, R. Cai, S.Q. Yang. A tennis video indexing approach through pattern discovery in interactive process. Pacific-Rim Conference on Multimedia, vol. 3331, pp. 49--56, 2004.
• 23: L. Xing, H. Yu, Q. Huang, Q. Ye, A. Divakaran. Subjective evaluation criterion for selecting affective features and modeling highlights. SPIE Conference on Multimedia Content Analysis and Management, vol. 6073, 2006.
• 24: G. Zhu, D. Liang, Y. Liu, Q. Huang, W. Gao. Improving particle filter with support vector regression for efficient visual tracking. IEEE International Conference on Image Processing, vol. 2, pp. 422--425, 2005.
• 25: B.K.P. Horn, B.G. Schunck. Determining optical flow. Artificial Intelligence, vol. 17, pp. 185--203, 1981.
• 26: Shuqiang Jiang , Qixiang Ye , Wen Gao , Tiejun Huang, A new method to segment playfield and its applications in match analysis in sports video, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027594]
• 27: Q. Ye, W. Gao, W. Zeng. Color image segmentation using density-based clustering. IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3, pp. 345--348, 2003.
• 28: Dorin Comaniciu , Visvanathan Ramesh , Peter Meer, Kernel-Based Object Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.5, p.564-575, May 2003 [doi>10.1109/TPAMI.2003.1195991]
• 29: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 30: Richard Hartley , Andrew Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, New York, NY, 2003
• 31: H. Liu, D. Zhou. Content-based news video story segmentation and video retrieval. SPIE Conference on Image and Graphics, vol. 4875, pp. 1038--1044, 2002.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Motion

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Tracking; Object recognition

General Terms:
Algorithms, Experimentation, Performance

Keywords:
action recognition, motion representation, object tracking, semantic analysis