article

Precision recall with user modeling (PRUM): Application to structured information retrieval

Authors:

G. DupretAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 25, Issue 1

Pages 1 - es

https://doi.org/10.1145/1198296.1198297

Published: 01 February 2007 Publication History

Abstract

Standard Information Retrieval (IR) metrics are not well suited for new paradigms like XML or Web IR in which retrievable information units are document elements and/or sets of related documents. Part of the problem stems from the classical hypotheses on the user models: They do not take into account the structural or logical context of document elements or the possibility of navigation between units. This article proposes an explicit and formal user model that encompasses a large variety of user behaviors. Based on this model, we extend the probabilistic precision-recall metric to deal with the new IR paradigms.

References

[1]

Baeza-Yates, R. and Ribeiro-Neto, B. 1999. Modern Information Retrieval. Addison Wesley, New York, NY.

Digital Library

[2]

Cleverdon, C. 1967. The cranfield tests on index language devices. In Proceedings Aslib. vol. 19, 173--192.

[3]

Cooper, W. S. 1968. Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. Amer. Documentat. 19, 30--41.

[4]

Cooper, W. S. 1973. On selecting a measure of retrieval effectiveness. part 1. J. Amer. Soci. Inform. Sci. 24, 87--100.

[5]

Dunlop, M. D. 1997. Time, relevance and interaction modeling for information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, 206--213.

Digital Library

[6]

Fuhr, N., Lalmas, M., and Malik, S., eds. 2003. INEX Proceedings.

[7]

Gövert, N., Kazai, G., Fuhr, N., and Lalmas, M. 2003. Evaluating the effectiveness of content-oriented XML retrieval. Tech. rep., Computer Science 6, University of Dortmund.

[8]

Heckerman, D. and Breese, J. S. 1994. A new look at causal independence. In Proceedings of the 10th Annual Conference on Uncertainty in Artificial Intelligence INEX Proceedings. (UAI'94). Morgan Kaufmann Publishers, San Francisco, CA, 286--292.

[9]

Kazai, G. 2003. Report on the INEX 2003 metrics group. INEX Proceedings. 184--190.

[10]

Kazai, G. and Lalmas, M. 2005. Notes on what to measure in inex. In Proceedings of the INEX Workshop on Element Retrieval Methodology, A. Trotman, M. Lalmas, and N. Fuhr, eds. INEX Proceedings. University of Otago, Univerisity of Glasgow. INEX Proceedings.

[11]

Kazai, G., Lalmas, M., and Piwowarski, B. 2003. Inex guidelines for topic development.

[12]

Kazai, G., Lalmas, M., and Vries, A. P. 2004. The overlap problem in content-oriented XML retrieval evaluation. In Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval. Sheffield, UK. ACM Press, 72--79.

Digital Library

[13]

Kekäläinen, J. and Järvelin, K. 2002. Using graded relevance assessments in IR evaluation. J. Ameri. Soc. Inform. Sci. 53, 13, 1120--1129.

Digital Library

[14]

Lalmas, M. and Moutogianni, E. 2000. A dempster-shafer indexing for the focussed retrieval of a hierarchically structured document space: Implementation and experiments on a web museum collection. In 6th RIAO Conference, Content-Based Multimedia Information Access. Paris, France.

[15]

Lawrence, P., Brin, S., Motwani, R., and Winograd, T. 1998. The pagerank citation ranking: Bringing order to the Web. Tech. rep., Stanford Digital Library Technologies Project.

[16]

Meadow, C. T., Kraft, D. H., and Boyce, B. R. 1999. Text Information Retrieval Systems. Academic Press, Orlando, FL.

Digital Library

[17]

Piwowarski, B. and Gallinari, P. 2003. Expected ratio of relevant units: A measure for structured information retrieval. INEX Proceedings.

[18]

Quintana, Y., Kamel, M., and McGeachy, R. 1993. Formal methods for evaluating information retrieval in hypertext systems. In Proceedings of the 11th Annual International Conference on Systems Documentation. Kitchener-Waterloo, Ontario, Canada. ACM Press, 259--272.

Digital Library

[19]

Raghavan, V. V., Jung, G. S., and Bollmann, P. 1989. A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inform. Syst. 7, 3, 205--229.

Digital Library

[20]

Saporta, G. 1990. Probabilités, analyse des données et statistique. Editions Technip, Paris, France.

[21]

Van Rijsbergen, C. J. 1979. Information Retrieval. Butterworths.

Digital Library

[22]

Voorhees, E. M. 2003. Common evaluation measures. The 12th Text Retrieval Conference (TREC'03). Number SP 500-255. NIST, 1--13.

[23]

Vries, A., Kazai, G., and Lalmas, M. 2004. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In Proceedings of RIAO (Recherche d'Information Assistée par Ordinateur (Computer Assisted Information Retrieval)). Avignon, France.

Cited By

Kusa WPeikos GStaudinger MLipani AHanbury AOosterhuis HBast HXiong C(2024)Normalised Precision at Fixed Recall for Evaluating TARProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672532(43-49)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672532
Pehcevski JPiwowarski B(2018)Evaluation Metrics for Structured Text RetrievalEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_152(1338-1348)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_152
Arour KYeferny T(2017)Formal concept analysis based user model for distributed systemsMultimedia Tools and Applications10.1007/s11042-016-3896-y76:15(16085-16105)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s11042-016-3896-y
Show More Cited By

Index Terms

Precision recall with user modeling (PRUM): Application to structured information retrieval
1. Information systems
  1. Information retrieval

Recommendations

Sound and complete relevance assessment for XML retrieval

In information retrieval research, comparing retrieval approaches requires test collections consisting of documents, user requests and relevance assessments. Obtaining relevance assessments that are as sound and complete as possible is crucial for the ...
High-Recall Information Retrieval from Linked Big Data
COMPSAC '15: Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference - Volume 02

In the current era of big data, high volumes of valuable information are available in collections of documents, the web, social networks, and high varieties of linked data. To search and retrieve useful information from these linked data, users often ...
A graphical user interface for the retrieval of hierarchically structured documents

Past research has proved that graphical user interfaces (GUIs) can significantly improve the effectiveness of the information access task. Our work is based on the consideration that structured document retrieval requires different user graphical ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 25, Issue 1

February 2007

153 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/1198296

Issue’s Table of Contents

Copyright © 2007 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2007

Published in TOIS Volume 25, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
1,203
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kusa WPeikos GStaudinger MLipani AHanbury AOosterhuis HBast HXiong C(2024)Normalised Precision at Fixed Recall for Evaluating TARProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672532(43-49)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672532
Pehcevski JPiwowarski B(2018)Evaluation Metrics for Structured Text RetrievalEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_152(1338-1348)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_152
Arour KYeferny T(2017)Formal concept analysis based user model for distributed systemsMultimedia Tools and Applications10.1007/s11042-016-3896-y76:15(16085-16105)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s11042-016-3896-y
Pehcevski JPiwowarski B(2017)Evaluation Metrics for Structured Text RetrievalEncyclopedia of Database Systems10.1007/978-1-4899-7993-3_152-2(1-12)Online publication date: 31-Jan-2017
https://doi.org/10.1007/978-1-4899-7993-3_152-2
Shin DKim C(2016)Query Extension of Retrieve System Using Hangul Word Embedding and AprioriThe Journal of Advanced Navigation Technology10.12673/jant.2016.20.6.61720:6(617-624)Online publication date: 31-Dec-2016
https://doi.org/10.12673/jant.2016.20.6.617
RAHIMI RSHAKERY ADADASHKARIMI JARIANNEZHAD MDEHGHANI MESFAHANI H(2016)Building a multi-domain comparable corpus using a learning to rank methodNatural Language Engineering10.1017/S135132491600016422:04(627-653)Online publication date: 15-Jun-2016
https://doi.org/10.1017/S1351324916000164
Najadat FKanaan G(2013)Approaches to Retrieve Verses of the Holy Quran Based on Full Meaning2013 Taibah University International Conference on Advances in Information Technology for the Holy Quran and Its Sciences10.1109/NOORIC.2013.68(310-315)Online publication date: Dec-2013
https://doi.org/10.1109/NOORIC.2013.68
Tahraoui MPinel-Sauvagnat KLaitang CBoughanem MKheddouci HNing L(2013)A survey on tree matching and XML retrievalComputer Science Review10.1016/j.cosrev.2013.02.0018(1-23)Online publication date: May-2013
https://doi.org/10.1016/j.cosrev.2013.02.001
Zhao Q(2013)Computational Awareness: Another Way towards IntelligenceComputational Intelligence10.1007/978-3-642-35638-4_1(3-14)Online publication date: 2013
https://doi.org/10.1007/978-3-642-35638-4_1
MENDES ACOHEUR L(2012)When the answer comes into question in question-answering: survey and open issuesNatural Language Engineering10.1017/S135132491100035019:01(1-32)Online publication date: 10-Jan-2012
https://doi.org/10.1017/S1351324911000350
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents