skip to main content
10.1145/2733373.2806279acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Analytic Quality: Evaluation of Performance and Insight in Multimedia Collection Analysis

Published: 13 October 2015 Publication History

Abstract

In this paper, we present analytic quality (AQ), a novel paradigm for the design and evaluation of multimedia analysis methods. AQ complements the existing evaluation methods based on either machine-driven benchmarks or user studies. AQ includes the notion of user insight gain and the time needed to acquire it, both critical aspects of large-scale multimedia collections analysis. To incorporate insight, AQ introduces a novel user model. In this model, each simulated user, or artificial actor, builds its insight over time, at any time operating with multiple categories of relevance. The methods are evaluated in timed sessions. The artificial actors interact with each method and steer the course by indicating relevant items throughout the session. AQ measures not only precision and recall, but also throughput, diversity of the results, and the accuracy of estimating the percentage of relevant items in the collection. AQ is shown to provide a wide picture of analytic capabilities of the evaluated methods and enumerate how their strengths differ for different purposes. The AQ time plots provide design suggestions for improving the evaluated methods. AQ is demonstrated to be more insightful than the classic benchmark evaluation paradigm both in terms of method comparison and suggestions for further design.

References

[1]
D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM MM, pages 223--232, 2013.
[2]
O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In ACM CIKM, pages 621--630, 2009.
[3]
T. Demeester, D. Trieschnigg, D. Nguyen, K. Zhou, and D. Hiemstra. Overview of the TREC 2014 federated Web search track. In TREC, 2014.
[4]
M. Eskevich, R. Aly, R. Ordelman, S. Chen, and G. J. F. Jones. The search and hyperlinking task at MediaEval 2013. In MediaEval, 2013.
[5]
M. Everingham, L. V. Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) challenge. IJCV, 88(2):303--338, 2010.
[6]
T. M. Green, W. Ribarsky, and B. Fisher. Building and applying a human cognition model for visual analytics. InfoVis, 8(1):1--13, Mar. 2009.
[7]
X.-S. Hua, Y. Ming, and J. Li. Mining knowledge from clicks: MSR-Bing image retrieval challenge. In IEEE ICMEW, pages 1--4, 2014.
[8]
D. P. Huijsmans and N. Sebe. How to complete performance graphs in content-based image retrieval: Add generality and normalize scope. IEEE TPAMI, 27(2), Feb. 2005.
[9]
B. Ionescu, M. Menéndez, H. Müller, and A. Popescu. Retrieving diverse social images at MediaEval 2013: Objectives, dataset, and evaluation. In MediaEval, 2013.
[10]
B. Ionescu, A.-L. Radu, M. Menéndez, H. Müller, A. Popescu, and B. Loni. Div400: A social image retrieval result diversification dataset. In ACM MMSys, pages 29--34, 2014.
[11]
A. Jaimes, N. Sebe, and D. Gatica-Perez. Human-centered computing: A multimedia perspective. In ACM MM, pages 855--864, 2006.
[12]
K. J\"arvelin and J. Kek\"al\"ainen. Cumulated gain-based evaluation of IR techniques. ACM TOIS, 20(4):422--446, Oct. 2002.
[13]
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675--678, 2014.
[14]
M. A. Larson, B. Ionescu, X. Anguera, M. Eskevich, P. Korshunov, M. Schedl, M. Soleymani, G. Petkos, R. F. E. Sutcliffe, J. Choi, and G. J. F. Jones, editors. Working Notes Proceedings of the MediaEval 2014 Workshop, 2014.
[15]
A. Lavie and A. Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In StatMT, pages 228--231.
[16]
M. Lestari Paramita, M. Sanderson, and P. Clough. Diversity in photo retrieval: Overview of the ImageCLEF photo task 2009. In Multilingual Information Access Evaluation II. Multimedia Experiments, pages 45--59. 2010.
[17]
Y. Li and B. Merialdo. VERT: Automatic evaluation of video summaries. In ACM MM, pages 851--854, 2010.
[18]
C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In ACL, pages 74--81, 2004.
[19]
A. Nenkova, R. Passonneau, and K. McKeown. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM TSLP, 4(2), May 2007.
[20]
C. North. Towards measuring visualization insight. IEEE TCGA, 26(3):6--9, 2006.
[21]
C. North, P. Saraiya, and K. Duca. A comparison of benchmark task and insight evaluation methods for information visualization. InfoVis, 10(3):162--181, July 2011.
[22]
P. Over, A. F. Smeaton, and G. Awad. The TRECVid 2008 BBC Rushes summarization evaluation. In ACM TRECVid Video Summarization Workshop, ACM MM, pages 1--20, 2008.
[23]
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL, pages 311--318, 2002.
[24]
W. A. Pike, J. Stasko, R. Chang, and T. A. O'Connell. The science of interaction. InfoVis, 8(4):263--274, 2009.
[25]
S. Rudinac, M. Larson, and A. Hanjalic. Learning crowdsourced user preferences for visual summarization of image collections. IEEE TMM, 15(6), Oct. 2013.
[26]
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575, 2014.
[27]
M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In ACM SIGIR, pages 555--562, 2010.
[28]
K. Schoeffmann. A user-centric media retrieval competition: The video browser showdown 2012--2014. IEEE Multimedia, 21(4):8--13, 2014.
[29]
M. D. Smucker and C. L. A. Clarke. Time-based calibration of effectiveness measures. In ACM SIGIR, pages 95--104, 2012.
[30]
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv:1409.4842, 2014.
[31]
J. J. Thomas and K. A. Cook. Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Computer Society, 2005.
[32]
B. Thomée and M. S. Lew. Interactive search in image retrieval: a survey. Int J of MIR, 1(2):71--86, July 2012.
[33]
S. Tong and E. Chang. Support vector machine active learning for image retrieval. In ACM MM, pages 107--118, 2001.
[34]
P. van der Corput and J. J. van Wijk. Effects of presentation mode and pace control on performance in image classification. IEEE TVCG, 20(12):2301--2309, Dec. 2014.
[35]
R. Vedantam, C. L. Zitnick, and D. Parikh. CIDEr: Consensus-based image description evaluation. arXiv:1411.5726, 2015.
[36]
R. Rehrurek and P. Sojka. Software framework for topic modelling with large corpora. In LREC, pages 45--50, 2010.
[37]
J. Zahálka, S. Rudinac, and M. Worring. New Yorker Melange: Interactive brew of personalized venue recommendations. In ACM MM, pages 205--208, 2014.
[38]
J. Zahálka and M. Worring. Towards interactive, intelligent, and integrated multimedia analytics. In IEEE VAST, pages 3--12, 2014.

Cited By

View all
  • (2024)Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language ModelsProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658032(978-987)Online publication date: 30-May-2024
  • (2024)Leveraging Query Expansion and Reformulation for Image Retrieval With Large Language and Vision-Language Models2024 International Conference on Content-Based Multimedia Indexing (CBMI)10.1109/CBMI62980.2024.10859227(1-7)Online publication date: 18-Sep-2024
  • (2024)Exquisitor at the Video Browser Showdown 2024: Relevance Feedback Meets Conversational SearchMultiMedia Modeling10.1007/978-3-031-53302-0_31(347-355)Online publication date: 29-Jan-2024
  • Show More Cited By

Index Terms

  1. Analytic Quality: Evaluation of Performance and Insight in Multimedia Collection Analysis

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '15: Proceedings of the 23rd ACM international conference on Multimedia
    October 2015
    1402 pages
    ISBN:9781450334594
    DOI:10.1145/2733373
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 October 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. evaluation
    2. interactivity
    3. multimedia search and exploration
    4. user insight

    Qualifiers

    • Research-article

    Funding Sources

    • Dutch Technology Foundation (STW)

    Conference

    MM '15
    Sponsor:
    MM '15: ACM Multimedia Conference
    October 26 - 30, 2015
    Brisbane, Australia

    Acceptance Rates

    MM '15 Paper Acceptance Rate 56 of 252 submissions, 22%;
    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)12
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language ModelsProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658032(978-987)Online publication date: 30-May-2024
    • (2024)Leveraging Query Expansion and Reformulation for Image Retrieval With Large Language and Vision-Language Models2024 International Conference on Content-Based Multimedia Indexing (CBMI)10.1109/CBMI62980.2024.10859227(1-7)Online publication date: 18-Sep-2024
    • (2024)Exquisitor at the Video Browser Showdown 2024: Relevance Feedback Meets Conversational SearchMultiMedia Modeling10.1007/978-3-031-53302-0_31(347-355)Online publication date: 29-Jan-2024
    • (2022)Influence of Late Fusion of High-Level Features on User Relevance Feedback for VideosProceedings of the 2nd International Workshop on Interactive Multimedia Retrieval10.1145/3552467.3554795(17-24)Online publication date: 14-Oct-2022
    • (2022)Evaluating a Bayesian-like relevance feedback model with text-to-image search initializationMultimedia Tools and Applications10.1007/s11042-022-14046-w82:15(22305-22341)Online publication date: 4-Nov-2022
    • (2021)Impact of Interaction Strategies on User Relevance FeedbackProceedings of the 2021 International Conference on Multimedia Retrieval10.1145/3460426.3463663(590-598)Online publication date: 24-Aug-2021
    • (2021)II-20: Intelligent and pragmatic analytic categorization of image collectionsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.303038327:2(422-431)Online publication date: Feb-2021
    • (2021)Search and Explore Strategies for Interactive Analysis of Real-Life Image Collections with Unknown and Unique CategoriesMultiMedia Modeling10.1007/978-3-030-67835-7_21(244-255)Online publication date: 21-Jan-2021
    • (2020)Exquisitor at the Lifelog Search Challenge 2020Proceedings of the Third Annual Workshop on Lifelog Search Challenge10.1145/3379172.3391718(19-22)Online publication date: 9-Jun-2020
    • (2019)Interactive Search and Exploration in Discussion Forums Using Multimodal EmbeddingsMultiMedia Modeling10.1007/978-3-030-37734-2_32(388-399)Online publication date: 24-Dec-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media