Crawlability
Crawlability shows which AI crawlers can read your website and which are shut out. LiveMentions reads your robots.txt, checks it against a catalog of AI bots, and gives every bot a status of Allowed, Partly restricted or Blocked, with the exact rule that decides it.
You find it in the sidebar, above AI Referrals. It uses no AI credits and is available on every active plan.
Why Crawlability matters
AI search and answer features use their own crawlers. A bot that is blocked cannot read your pages, so those pages cannot be used in its answers.
robots.txt files are usually written once and forgotten. They get edited by plugins, copied from other sites, or changed by a CDN or a security tool. A single line such as Disallow: / under the wrong user agent can shut a whole AI engine out, and nothing looks broken. The first sign is often that AI never mentions your brand.
Crawlability makes this visible in one screen, so you can open the door before you invest in content.
What robots.txt is
robots.txt is the file at yoursite.com/robots.txt that tells crawlers which parts of a website they may visit. Search engines have always followed it, and the AI crawlers published by the AI companies state that they follow it too. The page shows each vendor's own statement for every bot.
Crawlability shows what the file asks of each bot. It does not show which bots visited your site, because that comes from server logs.
The three tabs
- AI bots. Every bot, its status and the rule behind it. This article covers this tab.
- URL tester. Check one address on your website for every bot. See Crawlability: the URL tester and competitors.
- Competitors. Compare your robots.txt with your tracked competitors. See the same article.
What the statuses mean
Each bot gets one status, decided by the standard robots.txt rules: the bot's own group beats the rules for everyone, the longest matching rule wins, and wildcards work.
- Allowed. Your homepage and the pages checked are open to the bot. A rule that shuts out only a few pages is shown as a note and does not change the status. For example, rules for everyone that hide an admin area leave a bot Allowed, and the page lists the paths and how many pages they touch.
- Partly restricted. The bot can read your homepage, but real pages are shut out. This applies to any page when the file has rules written for that bot by name, and to a tenth or more of the pages checked when the bot only follows the rules for everyone.
- Blocked. Your homepage itself is shut out for that bot.
Every row shows the deciding rule, for example User-agent: GPTBot followed by Disallow: /. It also shows whether the bot has rules of its own or follows the rules for everyone, a crawl delay when there is one, and the paths its rules disallow.
How LiveMentions reads your robots.txt
- LiveMentions downloads robots.txt from your own website. This is a plain download and uses no AI. The page shows when the file was read, from which address and with which server answer.
- If the file names a sitemap, LiveMentions reads it and checks the pages against the rules. The page shows how many pages were checked.
- Each bot gets its status from the file and those pages.
If robots.txt cannot be read, for example because of a timeout, a server error or a web page returned instead of the file, the page says Could not read robots.txt and gives the reason. No bot gets a status until the file is readable. Only a real file not found means no rules, which means every bot is allowed, and the page says exactly that. LiveMentions never shows Allowed as a guess.
Training, search and user bots
Each bot has a type.
- Training bots collect pages to train AI models.
- Search bots find pages to answer questions.
- User bots fetch a page because a person asked for it.
Blocking a training bot is a legitimate choice and is never shown as an error. Blocking a search or user bot can keep an engine from using your pages in answers, and the page says so in careful words, for example "This can keep ChatGPT from using your pages".
Bots from ChatGPT, Claude, Gemini, Perplexity, Meta, Apple, Amazon, Microsoft, Mistral, Cohere, Common Crawl and others are covered, and a second card lists more verified bots. Each bot carries an honest label. Vendor-confirmed means the name, type and vendor come from the vendor's own documentation. Reported means only third-party bot directories list it, and the page shows a Reported marker and the type as Unconfirmed instead of guessing.
What is on the AI bots tab
- The stat strip: Fully open, With restrictions, Pages checked and Last checked.
- A file line with the address, the server answer and the size of robots.txt.
- Filters for Platform, Bot type and Status, and a search box.
- Two groups: bots with their own rules in robots.txt, and bots following the global rules for everyone.
- Other bots named in your file, listed with the name and rule only, so nothing in your file is hidden.
Click any bot to open a side panel with four cards: On your site, What it is, Does it follow robots.txt? and Source. The third card gives the vendor's own statement: it follows robots.txt, it may ignore it when a person asked for the page, it is a control token with no crawler, or it uses a different file.
The buttons
- Reload robots.txt reads your file again. Editors and above can use it.
- History lists each change to your file and which bots changed status, each described against the one before.
- View robots.txt shows the text as it was last read.
- Let AI search bots in gives you a ready-made robots.txt text for the AI search bots, built from the bots their vendors document.
- Export saves the table as CSV, XLSX or JSON.
Letting AI search bots in
LiveMentions does not change your website. To open the door:
- Press Let AI search bots in.
- Copy the text it gives you into your robots.txt and upload the file.
- Press Reload robots.txt to confirm the new statuses.
Plans, roles and credits
Crawlability is included on every active plan: trial, Starter, Growth, Scale and every lifetime plan. It uses no AI credits. All roles can look at the page. Editors and above can reload robots.txt, test an address and refresh competitors.
Ask your AI assistant
If you have connected an AI tool through MCP, it can read the same information with the get_crawlability tool, for example Is any AI search bot blocked on my site? See Connecting AI tools via MCP.
What Crawlability does not claim
- It shows what robots.txt asks of each bot. Allowed means your file allows the bot, not that the bot visited.
- It does not decide whether blocking a training bot is right for your business. That is your choice.
- It does not change your website. The text you copy is a suggestion that you or your developer applies.