Optimize

Understanding bot statuses (Allowed, Partly restricted, Blocked)

Every AI bot on the Crawlability page has one status, decided from your robots.txt.

Allowed

Your homepage and the pages checked are open to the bot. A rule that shuts out only a few pages is shown as a note and does not change the status. For example, rules for everyone that hide an admin area leave a bot Allowed, and the page lists the paths and how many pages they touch.

Partly restricted

The bot can read your homepage, but real pages are shut out. This applies to any page when your file has rules written for that bot by name, and to a tenth or more of the pages checked when the bot only follows the rules for everyone.

Blocked

Your homepage itself is shut out for that bot.

The rule behind each status

Every row shows the deciding rule, for example User-agent: GPTBot followed by Disallow: /. It also shows whether the bot has rules of its own or follows the rules for everyone.

When there is no status

If robots.txt cannot be read, no bot gets a status. The page says Could not read robots.txt and gives the reason.

For more, see Crawlability.