| robots.txt | |
| Long Name: | Robots Exclusion Protocol |
| Status: | Proposed Standard |
| First Published: | 1994 published, formally standardized in 2022 |
| Website: | , |
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with security through obscurity. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate server overload. In the 2020s many websites began denying bots that collect information for generative artificial intelligence.
The "robots.txt" file can be used in conjunction with sitemaps, another robot inclusion standard for websites.
The standard was proposed by Martijn Koster,[1] [2] when working for Nexor[3] in February 1994[4] on the www-talk mailing list, the main communication channel for WWW-related activities at the time. Charles Stross claims to have provoked Koster to suggest robots.txt, after he wrote a badly behaved web crawler that inadvertently caused a denial-of-service attack on Koster's server.[5]
The standard, initially RobotsNotWanted.txt, allowed web developers to specify which bots should not access their website or which pages bots should not access. The internet was small enough in 1994 to maintain a complete list of all bots; server overload was a primary concern. By June 1994 it had become a de facto standard;[6] most complied, including those operated by search engines such as WebCrawler, Lycos, and AltaVista.[7]
On July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under Internet Engineering Task Force.[8] A proposed standard was published in September 2022 as RFC 9309.
When a site owner wishes to give instructions to web robots they place a text file called in the root of the web site hierarchy (e.g.