Article

Hits hits TREC: exploring IR evaluation results with network analysis

Authors:

Stefano Mizzaro,

Stephen RobertsonAuthors Info & Claims

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 479 - 486

https://doi.org/10.1145/1277741.1277824

Published: 23 July 2007 Publication History

Abstract

We propose a novel method of analysing data gathered fromTREC or similar information retrieval evaluation experiments. We define two normalized versions of average precision, that we use to construct a weighted bipartite graph of TREC systems and topics. We analyze the meaning of well known - and somewhat generalized - indicators fromsocial network analysis on the Systems-Topics graph. We apply this method to an analysis of TREC 8 data; amongthe results, we find that authority measures systems performance, that hubness of topics reveals that some topics are better than others at distinguishing more or less effective systems, that with current measures a system that wants to be effective in TREC needs to be effective on easy topics, and that by using different effectiveness measures this is no longer the case.

References

[1]

M. Agosti, M. Bacchin, N. Ferro, and M. Melucci. Improving the automatic retrieval of text documents. In Proceedings of the 3rd CLEF Workshop, volume 2785 of LNCS, pages 279--290, 2003.

[2]

C. Buckley and E. Voorhees. Evaluating evaluation measure stability. In 23rd SIGIR, pages 33--40, 2000.

Digital Library

[3]

S. Chakrabarti. Mining the Web. Morgan Kaufmann, 2003.

[4]

G. V. Cormack and T. R. Lynam. Statistical precision of information retrieval evaluation. In 29th SIGIR, pages 533--540, 2006.

Digital Library

[5]

J. Kleinberg. Authoritative sources in a hyperlinked environment. J. of the ACM, 46(5):604--632, 1999.

Digital Library

[6]

M. Levene. An Introduction to Search Engines and Web Navigation. Addison Wesley, 2006.

Digital Library

[7]

L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank Citation Ranking: Bringing Order to the Web, 1998. http://dbpubs.stanford.edu:8090/pub/1999--66.

[8]

S. Robertson. On GMAP - and other transformations. In 13th CIKM, pages 78--83, 2006.

Digital Library

[9]

M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In 28th SIGIR, pages 162--169, 2005. http://doi.acm.org/10.1145/1076034.1076064.

Digital Library

[10]

I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In 24th SIGIR, pages 66--73, 2001.

Digital Library

[11]

TREC Common Evaluation Measures, 2005. http://trec.nist.gov/pubs/trec14/appendices/ CE.MEASURES05.pdf (Last visit: Jan. 2007).

[12]

Text REtrieval Conference (TREC). http://trec.nist.gov/ (Last visit: Jan. 2007).

[13]

E. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In 25th SIGIR, pages 316--323, 2002.

Digital Library

[14]

E. M. Voorhees. Overview of the TREC 2005 Robust Retrieval Track. In TREC 2005 Proceedings, 2005.

[15]

E. M. Voorhees and D. K. Harman. TREC - Experiment and Evaluation in Information Retrieval. MIT Press, 2005.

Digital Library

[16]

S. Wasserman and K. Faust. Social Network Analysis. Cambridge University Press, Cambridge, UK, 1994.

Cited By

Mothe JUllah M(2023)Selective Query Processing: A Risk-Sensitive Selection of Search ConfigurationsACM Transactions on Information Systems10.1145/360847442:1(1-35)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3608474
Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Otero DParapar JFerro NFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication MethodsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614916(1960-1970)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614916
Show More Cited By

Index Terms

Hits hits TREC: exploring IR evaluation results with network analysis
1. Information systems
  1. Information retrieval

Recommendations

On Using Fewer Topics in Information Retrieval Evaluations
ICTIR '13: Proceedings of the 2013 Conference on the Theory of Information Retrieval

The possibility of using fewer topics in TREC, and in TREC-like initiatives, has been studied recently, with encouraging results: even when decreasing consistently the number of topics (for example, using a topic subset of cardinality only 10, in place ...
Liberal relevance criteria of TREC -: counting on negligible documents?
SIGIR '02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval

Most test collections (like TREC and CLEF) for experimental research in information retrieval apply binary relevance assessments. This paper introduces a four-point relevance scale and reports the findings of a project in which TREC-7 and TREC-8 ...
Using crowdsourcing for TREC relevance assessment

Crowdsourcing has recently gained a lot of attention as a tool for conducting different kinds of relevance evaluations. At a very high level, crowdsourcing describes outsourcing of tasks to a large group of people instead of assigning such tasks to an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
788
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mothe JUllah M(2023)Selective Query Processing: A Risk-Sensitive Selection of Search ConfigurationsACM Transactions on Information Systems10.1145/360847442:1(1-35)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3608474
Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Otero DParapar JFerro NFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication MethodsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614916(1960-1970)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614916
Mothe J(2022)Analytics Methods to Understand Information Retrieval Effectiveness—A SurveyMathematics10.3390/math1012213510:12(2135)Online publication date: 19-Jun-2022
https://doi.org/10.3390/math10122135
Roitero KMaddalena EMizzaro SScholer F(2022)On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10268858:6Online publication date: 22-Apr-2022
https://dl.acm.org/doi/10.1016/j.ipm.2021.102688
Mothe JUllah MDemartini GZuccon GCulpepper JHuang ZTong H(2021)Defining an Optimal Configuration Set for Selective Search Strategy - A Risk-Sensitive ApproachProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482422(1335-1345)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482422
Gienapp LStein BHagen MPotthast Md'Aquin MDietze SHauff CCurry ECudre Mauroux P(2020)Estimating Topic Difficulty Using Normalized Discounted Cumulated GainProceedings of the 29th ACM International Conference on Information & Knowledge Management10.1145/3340531.3412109(2033-2036)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3340531.3412109
Ferro NSanderson MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Improving the Accuracy of System Performance Estimation by Using ShardsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3338062(805-814)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3338062
Zampieri FRoitero KCulpepper JKurland OMizzaro SPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)On Topic Difficulty in IR EvaluationProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331279(909-912)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331279
Roitero KCulpepper JSanderson MScholer FMizzaro S(2019)Fewer topics? A million topics? Both?! On topics subsets in test collectionsInformation Retrieval Journal10.1007/s10791-019-09357-wOnline publication date: 8-May-2019
https://doi.org/10.1007/s10791-019-09357-w
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten