robots.txt (Q80776)
standard used to advise web crawlers and scrapers not to index a web page or site
Language:
other details
aliases robots.txt protocol
 Robots Exclusion Protocol
description standard used to advise web crawlers and scrapers not to index a web page or site

External Links
(P646) /m/0psn8
(P3381) Robots.txt
Robots_Exclusion_Standard
(P3417) Robots-Exclusion-Protocol
robots-txt
(P3553) 19600309
(P6366) 163978704
(P8205) Glossary/Robots.txt
(P12946) robots.txt