Article

Estimating collection size with logistic regression

Authors:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 789 - 790

https://doi.org/10.1145/1277741.1277910

Published: 23 July 2007 Publication History

Get Access

Abstract

Collection size is an important feature to represent the content summaries of a collection, and plays a vital role in collection selection for distributed search. In uncooperative environments, collection size estimation algorithms are adopted to estimate the sizes of collections with their search interfaces. This paper proposes heterogeneous capture (HC) algorithm, in which the capture probabilities of documents are modeled with logistic regression. With heterogeneous capture probabilities, HC algorithm estimates collection size through conditional maximum likelihood. Experimental results on real web data show that our HC algorithm outperforms both multiple capture-recapture and capture history algorithms.

References

[1]

J. Alho. Logistic regression in capture--recapture models. Biometrics, 1990, vol. 46, 623--635.

Crossref

Google Scholar

[2]

Z. Bar-Yossef, M. Gurevich. Random sampling from a search engine's index. In Proceedings of WWW'06, 2006, 367--376.

Digital Library

Google Scholar

[3]

R. Huggins. On the statistical analysis of capture experiments. Biometrika, 1989, vol. 76, 133--140.

Crossref

Google Scholar

[4]

K. Liu, C. Yu, W. Meng. Discovering the representative of a search engine. In Proceedings of CIKM'02, 2002, 652--654.

Digital Library

Google Scholar

[5]

M. Shokouhi, J. Zobel, F. Scholer, S. Tahaghoghi. Capturing collection size for distributed non-cooperative retrieval. In Proceedings of SIGIR'06, 2006, 316--323.

Digital Library

Google Scholar

[6]

L. Si, J. Callan. Relevant document distribution estimation method for resource selection. In Proceedings of SIGIR'03, 2003, 298--305.

Digital Library

Google Scholar

Cited By

View all

Duarte Torres SHiemstra DHuibers TDownie JMcDonald RCole TSanderson RShipman F(2013)Vertical selection in the information domain of childrenProceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries10.1145/2467696.2467714(57-66)Online publication date: 22-Jul-2013
https://dl.acm.org/doi/10.1145/2467696.2467714
Khelghati MHiemstra Dvan Keulen MPardede E(2012)Size estimation of non-cooperative data collectionsProceedings of the 14th International Conference on Information Integration and Web-based Applications & Services10.1145/2428736.2428774(239-246)Online publication date: 3-Dec-2012
https://dl.acm.org/doi/10.1145/2428736.2428774
Kelly WStumpf M(2011)Assessing Coverage of Protein Interaction Data Using Capture–Recapture ModelsBulletin of Mathematical Biology10.1007/s11538-011-9680-274:2(356-374)Online publication date: 26-Aug-2011
https://doi.org/10.1007/s11538-011-9680-2
Show More Cited By

Index Terms

Estimating collection size with logistic regression
1. Information systems
  1. Information retrieval

Recommendations

Capturing collection size for distributed non-cooperative retrieval
SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval

Modern distributed information retrieval techniques require accurate knowledge of collection size. In non-cooperative environments, where detailed collection statistics are not available, the size of the underlying collections must be estimated. While ...
Age-based garbage collection

Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Age-based garbage collection
OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications

Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...

Comments

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
377
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Duarte Torres SHiemstra DHuibers TDownie JMcDonald RCole TSanderson RShipman F(2013)Vertical selection in the information domain of childrenProceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries10.1145/2467696.2467714(57-66)Online publication date: 22-Jul-2013
https://dl.acm.org/doi/10.1145/2467696.2467714
Khelghati MHiemstra Dvan Keulen MPardede E(2012)Size estimation of non-cooperative data collectionsProceedings of the 14th International Conference on Information Integration and Web-based Applications & Services10.1145/2428736.2428774(239-246)Online publication date: 3-Dec-2012
https://dl.acm.org/doi/10.1145/2428736.2428774
Kelly WStumpf M(2011)Assessing Coverage of Protein Interaction Data Using Capture–Recapture ModelsBulletin of Mathematical Biology10.1007/s11538-011-9680-274:2(356-374)Online publication date: 26-Aug-2011
https://doi.org/10.1007/s11538-011-9680-2
Shokouhi MZobel J(2009)Robust result merging using sample-based score estimatesACM Transactions on Information Systems (TOIS)10.1145/1508850.150885227:3(1-29)Online publication date: 19-May-2009
https://dl.acm.org/doi/10.1145/1508850.1508852
Lu JLi D(2009)Estimating deep web data source size by capture–recapture methodInformation Retrieval10.1007/s10791-009-9107-y13:1(70-95)Online publication date: 13-Aug-2009
https://doi.org/10.1007/s10791-009-9107-y

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Capturing collection size for distributed non-cooperative retrieval

Age-based garbage collection

Age-based garbage collection

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations