skip to main content
10.1145/1390334.1390448acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance

Published: 20 July 2008 Publication History

Abstract

Modeling the beyond-topical aspects of relevance are currently gaining popularity in IR evaluation. For example, the discounted cumulated gain (DCG) measure implicitly models some aspects of higher-order relevance via diminishing the value of relevant documents seen later during retrieval (e.g., due to information cumulated, redundancy, and effort). In this paper, we focus on the concept of negative higher-order relevance (NHOR) made explicit via negative gain values in IR evaluation. We extend the computation of DCG to allow negative gain values, perform an experiment in a laboratory setting, and demonstrate the characteristics of NHOR in evaluation. The approach leads to intuitively reasonable performance curves emphasizing, from the user's point of view, the progression of retrieval towards success or failure. We discuss normalization issues when both positive and negative gain values are allowed and conclude by discussing the usage of NHOR to characterize test collections.

References

[1]
Cleverdon, C. W. (1991) The Significance of the Cranfield Tests on Index Languages. In Proceedings of the 14th ACM SIGIR, 3--12. http://delivery.acm.org/
[2]
Cooper, W.S. (1973) On Selecting a Measure of Retrieval Effectiveness. Part 1. Journal of the American Society for Information Science 24(2), 87--100.
[3]
Cosijn, E. & Ingwersen, P. (2000) Dimensions of Relevance. Information Processing & Management, 36 (4), 533--550.
[4]
Järvelin, K. & Kekääläinen, J. (2002) Cumulated Gain-Based Evaluation of IR Techniques. ACM TOIS 20 (4), 422--446.
[5]
Kekäläinen, J. & Jäärvelin, K. (2002) Evaluating Information Retrieval Systems under the Challenges of Interaction and Multidimensional Dynamic Relevance. In Proceedings of the 4th CoLIS Conference, 253--270.
[6]
Korfhage, R. R. (1997), Information Storage and Retrieval, Wiley, New York, 1997, 349 p.
[7]
Mizzaro, S. (1996) Relevance: The Whole (Hi)story. Journal of the American Society for Information Science 48(9), 810--832. http://www3.interscience.wiley.com/cgi-bin/fulltext/39757/ PDFSTART
[8]
Myaeng, S. H. & Korfhage, R. R. (1990) Integration of User Profiles: Models and Experiments in Information Retrieval. Information Processing & Management 26 (6), 719--738.
[9]
Price, S.L., Nielsen, M.L., Delcambre, L.M.L., & Vedsted, P. (2007) Semantic Components Enhance Retrieval of Domain-specific Documents. In Proceedings of the 16th ACM CIKM, 429--438. http://portal.acm.org
[10]
Sakai, T. (2007) On Penalising Late Arrival of Relevant Documents in Information Retrieval Evaluation with Graded Relevance. In The First International Workshop on Evaluating Information Access, May 15, 2007, Tokyo, Japan, 32--43.
[11]
Saracevic, T. (1975) Relevance: A Review of and a Framework for Thinking on the Notion in Information Science. Journal of the American Society for Information Science 26 (6), 321--343.
[12]
Saracevic, T. (1996) Relevance Reconsidered '96. In Proceedings of the 2nd CoLIS Conference, 201--218.
[13]
Saracevic, T. (2006) Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II. Advances in Librarianship, vol. 30, 2006. 3--71.
[14]
Sormunen, E. (2002). Liberal Relevance Criteria of TREC - Counting on Negligible Documents? In Proceedings of the 25th ACM SIGIR, 324--330.
[15]
Voorhees, E. M. (2001) Evaluation by Highly Relevant Documents. In Proceedings of the 24th ACM SIGIR, 74--82.
[16]
Voorhees, E. M. (2007) TREC: Continuing Information Retrieval's Tradition of Experimentation. Communications of the ACM 50 (11), 51--54. http://portal.acm.org
[17]
Yang, Y., Lad, A., Lao, N., Harpale, A., Kisiel, B., & Rogati, M., (2007) Utility-based Information Distillation Over Temporally Sequenced Documents. In Proceedings of the 30th ACM SIGIR, 31--38.

Cited By

View all
  • (2024)A Blueprint of IR Evaluation Integrating Task and User CharacteristicsACM Transactions on Information Systems10.1145/367516242:6(1-38)Online publication date: 1-Jul-2024
  • (2020)Online Primal-Dual Mirror Descent under Stochastic ConstraintsProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33921574:2(1-36)Online publication date: 12-Jun-2020
  • (2020)Staleness Control for Edge Data AnalyticsProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33921564:2(1-24)Online publication date: 12-Jun-2020
  • Show More Cited By

Index Terms

  1. Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
      July 2008
      934 pages
      ISBN:9781605581644
      DOI:10.1145/1390334
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 20 July 2008

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. discounted cumulated gain
      2. evaluation
      3. relevance
      4. test collections

      Qualifiers

      • Research-article

      Conference

      SIGIR '08
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 792 of 3,983 submissions, 20%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)3
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 19 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)A Blueprint of IR Evaluation Integrating Task and User CharacteristicsACM Transactions on Information Systems10.1145/367516242:6(1-38)Online publication date: 1-Jul-2024
      • (2020)Online Primal-Dual Mirror Descent under Stochastic ConstraintsProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33921574:2(1-36)Online publication date: 12-Jun-2020
      • (2020)Staleness Control for Edge Data AnalyticsProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33921564:2(1-24)Online publication date: 12-Jun-2020
      • (2020)Understanding (Mis)Behavior on the EOSIO BlockchainProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/33921554:2(1-28)Online publication date: 12-Jun-2020
      • (2019)Report on the Second Workshop on Exploitation of Social Media for Emergency Relief and Preparedness (SMERP 2018) at the Web Conference (WWW) 2018ACM SIGIR Forum10.1145/3308774.330880052:2(163-168)Online publication date: 17-Jan-2019
      • (2019)Report on the Eighth International Workshop on Location and the Web (LocWeb 2018)ACM SIGIR Forum10.1145/3308774.330879852:2(145-152)Online publication date: 17-Jan-2019
      • (2019)Report on GLARE 2018ACM SIGIR Forum10.1145/3308774.330879652:2(132-137)Online publication date: 17-Jan-2019
      • (2019)Report on the 12th Russian Summer School in Information Retrieval (RuSSIR 2018)ACM SIGIR Forum10.1145/3308774.330879052:2(100-104)Online publication date: 17-Jan-2019
      • (2019)Report on CLEF 2018ACM SIGIR Forum10.1145/3308774.330878552:2(72-82)Online publication date: 17-Jan-2019
      • (2019)Salton Award KeynoteACM SIGIR Forum10.1145/3308774.330878252:2(52-63)Online publication date: 17-Jan-2019
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media