robots.txt
(Q80776)
standard used to advise web crawlers and scrapers not to index a web page or site
standard used to advise web crawlers and scrapers not to index a web page or site
Language:
Current Data About
robots.txt
other details
aliases |
robots.txt protocol Robots Exclusion Protocol |
description | standard used to advise web crawlers and scrapers not to index a web page or site |
External Links
(P646) |
/m/0psn8
|
(P3381) |
Robots.txt
Robots_Exclusion_Standard |
(P3417) |
Robots-Exclusion-Protocol
robots-txt |
(P3553) |
19600309
|
(P6366) |
163978704
|
(P8205) |
Glossary/Robots.txt
|
(P12946) |
robots.txt
|