500/503 HTTP Status Code On Robots.txt Can Remove Your Site From Google Search
Gary Illyes from Google said on LinkedIn that if your server returns a 500/503 HTTP status code for an extended period of time for your robots.txt file, then Google may remove your site completely from Google Search.
This is even if the rest of your site is accessible and not returning a 500 or 503 status code.
It is not just a 500/503 HTTP status code that you need to worry about, it is also an issue if your site does those network timeout issues.
Again, it has to be for an “extended period of time,” which was not defined, but I assume it is more than just a day or two.
Gary wrote, “A robots.txt file that returns a 500/503 HTTP status code for an extended period of time will remove your site from search results, even if the rest of the site is accessible to Googlebot.” “Same goes for network timeouts,” Gary added.