Article

Evaluating relevant in context: document retrieval with a twist

Authors:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 749 - 750

https://doi.org/10.1145/1277741.1277890

Published: 23 July 2007 Publication History

Get Access

Abstract

The Relevant in Context retrieval task is document or article retrieval with a twist, where not only the relevant articles should be retrieved but also the relevant information within each article (captured by a set of XML elements) should be correctly identified. Our main research question is: how to evaluate the Relevant in Context task? We propose a generalized average precision measure that meets two main requirements: i) the score reflects the ranked list of articles inherent in the result list, and at the same time ii) the score also reflects how well the retrieved information per article (i.e., the set of elements) corresponds to the relevant information. The resulting measure was used at INEX 2006.

References

[1]

C. Clarke, J. Kamps, and M. Lalmas. INEX 2006 retrieval task and result submission specification. In N. Fuhr, M. Lalmas, and A. Trotman, editors, INEX 2006 Workshop Pre--Proceedings, pages 381--388, 2006.

Google Scholar

[2]

INEX. INitiative for the Evaluation of XML retrieval, 2007. http://inex.is.informatik.uni-duisburg.de/.

Google Scholar

[3]

J. Kekäläinen and K. Järvelin. Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology, 53:1120--1129, 2002.

Digital Library

Google Scholar

[4]

A. Tombros, B. Larsen, and S. Malik. Report on the INEX 2004 interactive track. SIGIR Forum, 39:43--49, 2005.

Digital Library

Google Scholar

Cited By

View all

Dietz LDalton J(2020)Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage RetrievalDatenbank-Spektrum10.1007/s13222-020-00334-yOnline publication date: 20-Mar-2020
https://doi.org/10.1007/s13222-020-00334-y
SzláVik ZTombros ALalmas M(2018)Summarisation of the logical structure of XML documentsInformation Processing and Management: an International Journal10.1016/j.ipm.2011.11.00248:5(956-968)Online publication date: 29-Dec-2018
https://dl.acm.org/doi/10.1016/j.ipm.2011.11.002
Norozi MArvola PHe QIyengar ANejdl WPei JRastogi R(2013)Selection fusion in semi-structured retrievalProceedings of the 22nd ACM international conference on Information & Knowledge Management10.1145/2505515.2505686(1291-1300)Online publication date: 27-Oct-2013
https://dl.acm.org/doi/10.1145/2505515.2505686
Show More Cited By

Index Terms

Evaluating relevant in context: document retrieval with a twist
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
2. Information systems
  1. Information retrieval
  2. Information systems applications
    1. Digital libraries and archives

Recommendations

Sound and complete relevance assessment for XML retrieval

In information retrieval research, comparing retrieval approaches requires test collections consisting of documents, user requests and relevance assessments. Obtaining relevance assessments that are as sound and complete as possible is crucial for the ...
Evaluating the effectiveness of content-oriented XML retrieval methods
Abstract
Content-oriented XML retrieval approaches aim at a more focused retrieval strategy: Instead of retrieving whole documents, document components that are exhaustive to the information need while at the same time being as specific as possible should ...
Extended structural relevance framework: a framework for evaluating structured document retrieval
Abstract
A structured document retrieval (SDR) system aims to minimize the effort users spend to locate relevant information by retrieving parts of documents. To evaluate the range of SDR tasks, from element to passage to tree retrieval, numerous task-...

Comments

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
330
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dietz LDalton J(2020)Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage RetrievalDatenbank-Spektrum10.1007/s13222-020-00334-yOnline publication date: 20-Mar-2020
https://doi.org/10.1007/s13222-020-00334-y
SzláVik ZTombros ALalmas M(2018)Summarisation of the logical structure of XML documentsInformation Processing and Management: an International Journal10.1016/j.ipm.2011.11.00248:5(956-968)Online publication date: 29-Dec-2018
https://dl.acm.org/doi/10.1016/j.ipm.2011.11.002
Norozi MArvola PHe QIyengar ANejdl WPei JRastogi R(2013)Selection fusion in semi-structured retrievalProceedings of the 22nd ACM international conference on Information & Knowledge Management10.1145/2505515.2505686(1291-1300)Online publication date: 27-Oct-2013
https://dl.acm.org/doi/10.1145/2505515.2505686
Ali MConsens MLalmas M(2012)Extended structural relevance framework: a framework for evaluating structured document retrievalInformation Retrieval10.1007/s10791-012-9192-115:6(558-590)Online publication date: 6-Mar-2012
https://doi.org/10.1007/s10791-012-9192-1
Arvola PVainio J(2011)The Potential Benefit of Focused Retrieval in Relevant-in-Context TaskComparative Evaluation of Focused Retrieval10.1007/978-3-642-23577-1_2(33-43)Online publication date: 2011
https://doi.org/10.1007/978-3-642-23577-1_2
Arvola PVainio J(2010)The potential benefit of focused retrieval in relevant-in-context taskProceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval10.5555/2040369.2040372(33-43)Online publication date: 13-Dec-2010
https://dl.acm.org/doi/10.5555/2040369.2040372
Ingwersen PLykke MBogers TLarsen BLund HBelkin NKelly D(2010)Assessors' search result satisfaction associated with relevance in a scientific domainProceedings of the third symposium on Information interaction in context10.1145/1840784.1840826(283-288)Online publication date: 18-Aug-2010
https://dl.acm.org/doi/10.1145/1840784.1840826
Arvola PKekäläinen JJunkkari M(2010)Expected reading effort in focused retrieval evaluationInformation Retrieval10.1007/s10791-010-9133-913:5(460-484)Online publication date: 6-May-2010
https://doi.org/10.1007/s10791-010-9133-9
Kamps JKoolen MLalmas MChua TLeong MMyaeng SOard DSebastiani F(2008)Locating relevant text within XML documentsProceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval10.1145/1390334.1390536(847-848)Online publication date: 20-Jul-2008
https://dl.acm.org/doi/10.1145/1390334.1390536

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Sound and complete relevance assessment for XML retrieval

Evaluating the effectiveness of content-oriented XML retrieval methods

Extended structural relevance framework: a framework for evaluating structured document retrieval

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations