People also ask
(discussed in more detail below) In order to be found, a robots.txt file must be placed in a website’s top-level directory. Robots.txt is case sensitive: the file must be named “robots.txt” (not Robots.txt, robots.TXT, or otherwise). Some user agents (robots) may choose to ignore your robots.txt file.
Here are a few examples of robots.txt in action for a www.example.com site: Using this syntax in a robots.txt file would tell all web crawlers not to crawl any pages on www.example.com, including the homepage. Using this syntax in a robots.txt file tells web crawlers to crawl all pages on www.example.com, including the homepage.
A search engine will cache the robots.txt contents, but usually updates the cached contents at least once a day. If you change the file and want to update it more quickly than is occurring, you can submit your robots.txt url to Google. So many robots!