Troubleshooting
Why is a bot Allowed when my robots.txt has a Disallow line?
A bot is Allowed when your Disallow rules shut out paths that touch none, or very few, of the pages checked. The rule is still shown: the page lists the paths and the number of pages as a note.
When the status stays Allowed
- The rule shuts out paths that none of your sitemap pages use, for example an admin area.
- The rule applies to everyone and affects less than a tenth of the pages checked.
When the status changes
- Partly restricted. Real pages are shut out: any page when your file has rules written for that bot by name, or a tenth or more of the pages checked when the bot only follows the rules for everyone.
- Blocked. Your homepage itself is shut out for that bot.
If you expected a different result
Open the bot's side panel and read On your site. It shows the deciding rule and the paths its rules disallow. Use the URL tester to check an exact address.
For more, see Crawlability.