When it comes to SEO, the robots.txt file plays a vital role that many SEOs don’t realize. It can negatively affect your SEO efforts and might stay under the radar unless you analyze the robots.txt file. According to Gary Illyes of Google, a 429 or 5xx error on robots.txt and important pages (e.g., the homepage) can cause your website to be deindexed from Google.
The HTTP Status Code of your robots.txt matters to Google.
1. Robots.txt is Unavailable: 4xx Status Code
2. Robots.txt is Unreachable: 5xx & 429 Status Code
Robots.txt Best Practices
Make sure you follow the best practices below:
a. Robots.txt is in the root folder (redirection works up to 5 hops)
Example: https://selfcanonical.com/robots.txt
b. Robots.txt is in lower case font
c. Robots.txt status code is 200 (OK)
Robots.txt Tester
- Robots.txt checker by Dave Smart – https://tamethebots.com/tools/robotstxt-checker
- Robots.txt checker by Real Robots Txt – https://www.realrobotstxt.com/
Recommended Resources:
- RFC 9309: Robots Exclusion Protocol
- Search Central Document: How Google interprets the robots.txt specification