skip to main content
10.1145/1148170.1148218acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
Article

Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM)

Published: 06 August 2006 Publication History

Abstract

Standard Information Retrieval (IR) metrics assume a simple model where documents are understood as independent units. Such an assumption is not adapted to new paradigms like XML or Web IR where retrievable informations are parts of documents or sets of related documents. Moreover, classical hypotheses assumes that the user ignores the structural or logical context of document elements and hence the possibility of navigation between units. EPRUM is a generalisation of Precision-Recall (PR) that aims at allowing the user to navigate or browse in the corpus structure. Like the Cumulated Gain metrics, it is able to handle continuous valued relevance. We apply and compare EPRUM in the context of XML Retrieval -- a very active field for evaluation metrics. We also explain how EPRUM can be used in other IR paradigms.

References

[1]
R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, New York, USA, 1999.
[2]
P. Billingsley. Probability and Measure. Wiley, New York, 1979.
[3]
C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40, New York, NY, USA, 2000. ACM Press.
[4]
C. Cleverdon. The cranfield tests on index language devices. In Aslib proceedings, volume 19, pages 173--192, 1967.
[5]
N. Fuhr, M. Lalmas, and S. Malik, editors. INEX 2003 Proceedings, 2003.
[6]
N. Fuhr, M. Lalmas, S. Malik, and G. Kazai, editors. INEX 2005 Proceedings, 2005.
[7]
N. Gövert, G. Kazai, N. Fuhr, and M. Lalmas. Evaluating the effectiveness of content-oriented XML retrieval. Technical report, University of Dortmund, Computer Science 6, 2003.
[8]
D. Heckerman and J. S. Breese. A new look at causal independence. In Proceedings of the Tenth Annual Conference on Uncertainty in Artificial Intelligence (UAI--94), pages 286--292, San Francisco, CA, 1994. Morgan Kaufmann Publishers.
[9]
G. Kazai. Report on the INEX 2003 metrics group. In Fuhr et al. {5}, pages 184--190.
[10]
G. Kazai and M. Lalmas. Inex 2005 evaluation metrics. In Fuhr et al. {6}.
[11]
G. Kazai, M. Lalmas, and A. P. Vries. The overlap problem in content-oriented XML retrieval evaluation. In Proceedings of the 27th annual international conference on Research and development in information retrieval, pages 72--79, Sheffield, UK, July 2004. ACM Press.
[12]
J. Kekäläinen and K. Järvelin. Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science (JASIS), 53(13):1120--1129, 2002.
[13]
P. Lawrence, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project, 1998.
[14]
B. Piwowarski and P. Gallinari. Expected ratio of relevant units: A measure for structured information retrieval. In Fuhr et al. {5}.
[15]
V. V. Raghavan, G. S. Jung, and P. Bollmann. A critical investigation of recall and precision as measures of retrieval system performance. ACM Transactions on Information Systems, 7(3):205--229, 1989.
[16]
S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, 33:294--304, 1977.
[17]
T. Saracevic. Relevance reconsidered. In Proceedings of the Second International Conference on Conceptions of Library and Information Science, volume 39, pages 201--218, Copenhagen, Danemark, 1996.
[18]
E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In SIGIR '98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia, pages 315--323. ACM, 1998.
[19]
E. M. Voorhees. Common evaluation measures. In The Twelfth Text Retrieval Conference (TREC 2003), number SP 500-255, pages 1--13. NIST, 2003.
[20]
A. Vries, G. Kazai, and M. Lalmas. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In Proceedings of RIAO (Recherche d'Information Assistée par Ordinateur (Computer Assisted Information Retrieval)), Avignon, France, Apr. 2004.
[21]
A. Woodley and S. Geva. Xcg overlap at inex 2004. In Fuhr et al. {6}.

Cited By

View all

Index Terms

  1. Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM)

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
    August 2006
    768 pages
    ISBN:1595933697
    DOI:10.1145/1148170
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 August 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. XML retrieval
    2. evaluation
    3. metric
    4. passage retrieval
    5. recall-precision
    6. web retrieval

    Qualifiers

    • Article

    Conference

    SIGIR06
    Sponsor:
    SIGIR06: The 29th Annual International SIGIR Conference
    August 6 - 11, 2006
    Washington, Seattle, USA

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)4
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2018)Evaluation Metrics for Structured Text RetrievalEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_152(1338-1348)Online publication date: 7-Dec-2018
    • (2017)Evaluation Metrics for Structured Text RetrievalEncyclopedia of Database Systems10.1007/978-1-4899-7993-3_152-2(1-12)Online publication date: 31-Jan-2017
    • (2016)A Secure JPEG Image Retrieval Method in Cloud EnvironmentCloud Computing and Security10.1007/978-3-319-48674-1_42(476-486)Online publication date: 1-Nov-2016
    • (2013)A linear and monotonic strategy to keyword search over RDF dataProceedings of the 13th international conference on Web Engineering10.1007/978-3-642-39200-9_28(338-353)Online publication date: 8-Jul-2013
    • (2012)Contextualization using hyperlinks and internal hierarchical structure of Wikipedia documentsProceedings of the 21st ACM international conference on Information and knowledge management10.1145/2396761.2396855(734-743)Online publication date: 29-Oct-2012
    • (2012)Semantic-Aware Metadata Organization Paradigm in Next-Generation File SystemsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2011.16923:2(337-344)Online publication date: 1-Feb-2012
    • (2012)Path-Oriented Keyword Search Query over RDFSemantic Search over the Web10.1007/978-3-642-25008-8_4(81-107)Online publication date: 28-Jan-2012
    • (2011)Efficient and effective ranking in Top-k exploration for keyword search on RDF2011 IEEE International Conference on Information Reuse & Integration10.1109/IRI.2011.6009522(66-70)Online publication date: Aug-2011
    • (2011)Path-oriented keyword search over graph-modeled Web dataWorld Wide Web10.1007/s11280-011-0153-115:5-6(631-661)Online publication date: 21-Dec-2011
    • (2010)Improving XML search by generating and utilizing informative result snippetsACM Transactions on Database Systems10.1145/1806907.180691135:3(1-45)Online publication date: 30-Jul-2010
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media