Crawlability shows which AI crawlers can read your website and which are shut out. LiveMentions reads your robots.txt, checks it against a catalog of AI bots, and shows every bot as Allowed, Partly restricted or Blocked, with the exact rule behind it.

Why it matters

A blocked AI bot cannot read your pages, so those pages cannot be used in its answers. One wrong line in robots.txt, often added by a plugin or a security setting, can shut a whole AI engine out without anything looking broken.

What robots.txt is

robots.txt is the file at yoursite.com/robots.txt that tells crawlers which parts of a website they may visit. The AI crawlers published by the AI companies state that they follow it.

Where to find it

Crawlability is in the sidebar, above AI Referrals. It has three tabs: AI bots, URL tester and Competitors. It uses no AI credits and is included on every active plan.

How to let AI search bots in

1. Press Let AI search bots in.
2. Copy the text into your robots.txt and upload the file.
3. Press Reload robots.txt to confirm the new statuses.

What it does not show

Crawlability shows what your robots.txt asks of each bot. It does not show which bots visited your site, because that comes from server logs.

For more, see Crawlability.