Article

A large-scale study of robots.txt

Authors:

Yang Sun,

Ziming Zhuang,

C. Lee GilesAuthors Info & Claims

WWW '07: Proceedings of the 16th international conference on World Wide Web

Pages 1123 - 1124

https://doi.org/10.1145/1242572.1242726

Published: 08 May 2007 Publication History

Get Access

Abstract

Search engines largely rely on Web robots to collect information from the Web. Due to the unregulated open-access nature of the Web, robot activities are extremely diverse. Such crawling activities can be regulated from the server side by deploying the Robots Exclusion Protocol in a file called robots.txt. Although it is not an enforcement standard, ethical robots (and many commercial) will follow the rules specified in robots.txt. With our focused crawler, we investigate 7,593 websites from education, government, news, and business domains. Five crawls have been conducted in succession to study the temporal changes. Through statistical analysis of the data, we present a survey of the usage of Web robots rules at the Web scale. The results also show that the usage of robots.txt has increased over time.

References

[1]

M. Drott. Indexing aids at corporate websites: The use of robots.txt and meta tags. Information Processing and Management, 38(2):209--219, 2002.

Digital Library

Google Scholar

[2]

B. Kelly and I. Peacock. Webwatching uk web communities: Final report for the webwatch project. British Library Research and Innovation Report, 1999.

Google Scholar

[3]

M. Koster. A method for web robots control. In the Internet Draft, The Internet Engineering Task Force (IETF), 1996.

Google Scholar

[4]

G. Pant, P. Srinivasan, and F. Menczer. Crawling the Web, chapter Web Dynamics. Springer-Verlag, 2004.

Google Scholar

Cited By

View all

Hegade PLingadhal NJain SKhan UVijeth K(2021)Crawler by Contextual InferenceSN Computer Science10.1007/s42979-021-00574-z2:3Online publication date: 16-Apr-2021
https://doi.org/10.1007/s42979-021-00574-z
Sisodia DKhandelwal RAnuragi A(2018)Categorization Performance of Unsupervised Learning Techniques for Web Robots Sessions2018 International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA.2018.8597200(1370-1374)Online publication date: Jul-2018
https://doi.org/10.1109/ICIRCA.2018.8597200
Sisodia DBorkar RShrawgi H(2018)Performance Evaluation of Large Data Clustering Techniques on Web Robot Session DataMachine Intelligence and Signal Analysis10.1007/978-981-13-0923-6_47(545-553)Online publication date: 8-Aug-2018
https://doi.org/10.1007/978-981-13-0923-6_47
Show More Cited By

A large-scale study of robots.txt
1. Information systems

Recommendations

A larger scale study of robots.txt
WWW '08: Proceedings of the 17th international conference on World Wide Web

A website can regulate search engine crawler access to its content using the robots exclusion protocol, specified in its robots.txt file. The rules in the protocol enable the site to allow or disallow part or all of its content to certain crawlers, ...
Measuring the web crawler ethics
WWW '10: Proceedings of the 19th international conference on World wide web

Web crawlers are highly automated and seldom regulated manually. The diversity of crawler activities often leads to ethical problems such as spam and service attacks. In this research, quantitative models are proposed to measure the web crawler ethics ...
BotSeer: An Automated Information System for Analyzing Web Robots
ICWE '08: Proceedings of the 2008 Eighth International Conference on Web Engineering

Robots.txt files are vital to the web since they are supposed to regulate what search engines can and cannot crawl. We present BotSeer, a Web-based information system and search tool that provides resources and services for researching Web robots and ...

Comments

Information & Contributors

Information

Published In

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

1382 pages

ISBN:9781595936547

DOI:10.1145/1242572

General Chairs:
Carey Williamson
University of Calgary, Canada
,
Mary Ellen Zurko
IBM, USA
,
Program Chairs:
Peter Patel-Schneider
Bell Labs Research, USA
,
Prashant Shenoy
University of Massachusetts at Amherst, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW'07

Sponsor:

WWW'07: 16th International World Wide Web Conference

May 8 - 12, 2007

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
788
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)8

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hegade PLingadhal NJain SKhan UVijeth K(2021)Crawler by Contextual InferenceSN Computer Science10.1007/s42979-021-00574-z2:3Online publication date: 16-Apr-2021
https://doi.org/10.1007/s42979-021-00574-z
Sisodia DKhandelwal RAnuragi A(2018)Categorization Performance of Unsupervised Learning Techniques for Web Robots Sessions2018 International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA.2018.8597200(1370-1374)Online publication date: Jul-2018
https://doi.org/10.1109/ICIRCA.2018.8597200
Sisodia DBorkar RShrawgi H(2018)Performance Evaluation of Large Data Clustering Techniques on Web Robot Session DataMachine Intelligence and Signal Analysis10.1007/978-981-13-0923-6_47(545-553)Online publication date: 8-Aug-2018
https://doi.org/10.1007/978-981-13-0923-6_47
Brown KDoran D(2017)Realistic Traffic Generation for Web Robots2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA)10.1109/ICMLA.2017.0-161(178-185)Online publication date: Dec-2017
https://doi.org/10.1109/ICMLA.2017.0-161
Cho YPan J(2015)Design and Implementation of Website Information Disclosure Assessment SystemPLOS ONE10.1371/journal.pone.011718010:3(e0117180)Online publication date: 13-Mar-2015
https://doi.org/10.1371/journal.pone.0117180
Lopez LDuerr RKhalsa S(2015)Optimizing apache nutch for domain specific crawling at large scaleProceedings of the 2015 IEEE International Conference on Big Data (Big Data)10.1109/BigData.2015.7363976(1967-1971)Online publication date: 29-Oct-2015
https://dl.acm.org/doi/10.1109/BigData.2015.7363976
Brunelle JKelly MSalahEldeen HWeigle MNelson M(2015)Not all mementos are created equal: measuring the impact of missing resourcesInternational Journal on Digital Libraries10.1007/s00799-015-0150-616:3-4(283-301)Online publication date: 6-May-2015
https://doi.org/10.1007/s00799-015-0150-6
Goodwin MCamacho DAkerkar RRodriguez Moreno M(2013)Towards automatic assessment of government web sitesProceedings of the 3rd International Conference on Web Intelligence, Mining and Semantics10.1145/2479787.2479805(1-12)Online publication date: 12-Jun-2013
https://dl.acm.org/doi/10.1145/2479787.2479805
Ceri SBozzon ABrambilla MDella Valle EFraternali PQuarteroni SCeri SBozzon ABrambilla MDella Valle EFraternali PQuarteroni S(2013)Search EnginesWeb Information Retrieval10.1007/978-3-642-39314-3_6(71-90)Online publication date: 2013
https://doi.org/10.1007/978-3-642-39314-3_6
Leung CChan AMilani ALiu JLi Y(2012)Intelligent Social Media Indexing and Sharing Using an Adaptive Indexing Search EngineACM Transactions on Intelligent Systems and Technology10.1145/2168752.21687613:3(1-27)Online publication date: 1-May-2012
https://dl.acm.org/doi/10.1145/2168752.2168761
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Recommendations

A larger scale study of robots.txt

Measuring the web crawler ethics

BotSeer: An Automated Information System for Analyzing Web Robots

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations