skip to main content
10.1145/1178677.1178722acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
Article

Evaluation campaigns and TRECVid

Published: 26 October 2006 Publication History

Abstract

The TREC Video Retrieval Evaluation (TRECVid)is an international benchmarking activity to encourage research in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations 1 interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video corpus,automatic detection of a variety of semantic and low-level video features, shot boundary detection and the detection of story boundaries in broadcast TV news. This paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, high-lighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation bench-marking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some of them in the paper concluding that on balance they have had a very positive impact on research progress.

References

[1]
AMI: Augmented Multi-Person Interaction. URL:www.amiproject.org/,Last checked 21 June 2006.
[2]
ARGOS: Evaluation Campaign for Surveillance Tools of Video Content. URL: www.irit.fr/recherches/SAMOVA/MEMBERS/-JOLY/argos/, Last checked 21 June 2006.
[3]
CLEAR '06 Evaluation Campaign and Workshop - Classification of Events, Activities and Relationships. URL:www.clear-evaluation.org/,Last checked 21 June 2006.
[4]
ETISEO: Video Understanding Evaluation. URL:www.silogic.fr/etiseo/, June 2006.
[5]
Face Recognition Grand Challenge. URL:www.frvt.org/FRGC,2006.
[6]
INEX:INitiative for the Evaluation of XML Retrieval. URL:inex.is.informatik.uni-duisburg.de/,Last checked 21 June 2006.
[7]
The Internet Archive Movie Archive home page. URL:www.archive.org/movies, 2006.
[8]
Lscom lexicon definitions and annotations. URL:www.ee.columbia.edu/dvmm/lscom, 2006.
[9]
NTCIR:NII Test Collection for IR Systems Project. URL:research.nii.ac.jp/ntcir/, Last checked June 2006.
[10]
PETS 2006: Ninth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance. URL:www.pets2006.net/,Last checked 21 June 2006.
[11]
The Benchathlon Network: Home of CBIR Benchmarking. URL:www.benchathlon.net/, Last checked 21 June 2006.
[12]
The Cross-Language Evaluation Forum (CLEF). URL:clef.isti.cnr.it/, Last checked 21 June 2006.
[13]
The IMAG-EVAL Evaluation Campaign. URL:www.imageval.org/, Last checked 21 June 2006.
[14]
M. G. Christel and A. G. Hauptmann. The Use and Utility of High-Level Semantic Features in Video Retrieval. In Proceedings of the International Conference on Video Retrieval pages 134--144, Singapore, 20-22 July 2005.
[15]
N. Fuhr and M. Lalmas. Introduction to the Special Issue on INEX. Information Retrieval 8(4):515--519, 2005.
[16]
A. G. Hauptmann and M. G. Christel. Successful Approaches in the TREC Video Retrieval Evaluations. In Proceedings of the 12th ACM International Conference on Multimedia pages 668--675, New York, NY, USA, 10-16 October 2004.
[17]
P. Ingwersen and K. Järvelin. The Turn: Integration of Information Seeking and Retrieval in Context Springer: the Kluwer International Series on Information Retrieval, 2005.
[18]
B. F. J. Manly. Randomization, Bootstrap, and Monte Carlo Methods in Biology Chapman & Hall, London, UK, 2nd edition, 1997.
[19]
M. R. Naphade and J. R. Smith. On the Detection ofSemantic Concepts at TRECVID.In Proceedings of the 12th ACM International Conference on Multimedia pages 660--667, New York, NY, USA, 10-16 October 2004.
[20]
NIST. TREC Video Retrieval Evaluation Publications. URL: www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html,2006.
[21]
R. Ruiloba, P. Joly, S. Marchand-Maillet, and G. Quénot. Towards a Standard Protocol for the Evaluation of Video-to-Shots Segmentation Algorithms. In European Workshop on Content Based Multimedia Indexing Toulouse, France, October 1999.
[22]
C. G. Snoek, M. Worring, J.-M. Geusebroek, D. C. Koelma, and F. J. Seinstra. On the surplus value of semantic video analysis beyond the key frame.In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME)July 2005.
[23]
C. G. Snoek, M. Worring, J.-M. Geusebroek, D. C. Koelma, F. J. Seinstra, and A. Smeulders. The semantic path finder: Using an authoring metaphor for generic multimedia indexing. IEEE Transactions, PAMI in press, 2006.

Cited By

View all
  • (2024)Nollywood Movie Sequences Summarization Dataset for Machine Learning ApplicationsBritish Journal of Computer, Networking and Information Technology10.52589/BJCNIT-CVZ6HABH7:2(43-48)Online publication date: 24-Jun-2024
  • (2024)Performance Evaluation in Multimedia RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367888121:1(1-23)Online publication date: 14-Oct-2024
  • (2024)Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept BankProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658052(73-82)Online publication date: 30-May-2024
  • Show More Cited By
  1. Evaluation campaigns and TRECVid

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval
    October 2006
    344 pages
    ISBN:1595934952
    DOI:10.1145/1178677
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 October 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. benchmarking
    2. evaluation
    3. video retrieval

    Qualifiers

    • Article

    Conference

    MM06
    MM06: The 14th ACM International Conference on Multimedia 2006
    October 26 - 27, 2006
    California, Santa Barbara, USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)53
    • Downloads (Last 6 weeks)9
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Nollywood Movie Sequences Summarization Dataset for Machine Learning ApplicationsBritish Journal of Computer, Networking and Information Technology10.52589/BJCNIT-CVZ6HABH7:2(43-48)Online publication date: 24-Jun-2024
    • (2024)Performance Evaluation in Multimedia RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367888121:1(1-23)Online publication date: 14-Oct-2024
    • (2024)Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept BankProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658052(73-82)Online publication date: 30-May-2024
    • (2024)A Survey of Video Datasets for Grounded Event Understanding2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00727(7314-7327)Online publication date: 17-Jun-2024
    • (2024)GreenScreen: A Multimodal Dataset for Detecting Corporate Greenwashing in the WildMultiMedia Modeling10.1007/978-3-031-56435-2_8(96-109)Online publication date: 20-Mar-2024
    • (2023)(Un)likelihood Training for Interpretable EmbeddingACM Transactions on Information Systems10.1145/363275242:3(1-26)Online publication date: 13-Nov-2023
    • (2023)A New Approach for Evaluating Movie SummarizationProceedings of the 2nd Workshop on User-centric Narrative Summarization of Long Videos10.1145/3607540.3617137(9-15)Online publication date: 29-Oct-2023
    • (2023)Video summarization using deep learning techniques: a detailed analysis and investigationArtificial Intelligence Review10.1007/s10462-023-10444-056:11(12347-12385)Online publication date: 15-Mar-2023
    • (2022)A Video Summarization Model Based on Deep Reinforcement Learning with Long-Term DependencySensors10.3390/s2219768922:19(7689)Online publication date: 10-Oct-2022
    • (2022)An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia RetrievalProceedings of the 2nd International Workshop on Interactive Multimedia Retrieval10.1145/3552467.3554797(33-39)Online publication date: 14-Oct-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media