Crawlability: the URL tester and competitors
Two tabs on the Crawlability page go beyond your robots.txt as a whole. The URL tester checks one exact address for every AI bot. The Competitors tab puts your robots.txt next to the robots.txt of the competitors you track.
The URL tester
Type an address on your own website and the URL tester shows, for each bot, whether robots.txt allows or blocks that exact address and which rule line decides it. Addresses on other websites are refused.
The live check
Under the address box, the option Also check how my site answers these bots goes one step further. For each key bot, LiveMentions makes one request to the address using that bot's user agent and reports the real server answer, for example 200 or 403.
This catches what robots.txt cannot show: a firewall or CDN that answers AI bots differently from a browser. A bot that robots.txt allows can still be turned away by a security setting. When the answer differs from a browser, the row says Different from a browser.
The live check is a simulation, and the page says so. Some firewalls check where a bot comes from, so a real crawler may be treated differently from the test request.
In LiveMentions' own tests the live check found a site whose firewall answered 403 to the ChatGPT and Claude bots while its robots.txt allowed them and a browser got 200.
The URL tester needs an editor role, because it makes real requests to your website.
The Competitors tab
The Competitors tab compares your robots.txt with the competitors you track on the Competitors page.
- A summary for each competitor shows how many bots it leaves fully open.
- A bot-by-bot grid shows You and the competitors you choose, side by side.
- The switch Only show bots treated differently hides every bot where you and the competitors you picked match, so the differences stand out.
Only each competitor's public robots.txt is read. The statuses mean what that file says about the homepage and its rules, and no sitemap is checked for competitors, which the page says. A competitor whose file cannot be read shows Could not read, never a status.
With no tracked competitors, the tab asks you to track some on the Competitors page. See Adding and managing competitors.
What to do with it
- Test the address of a new page before you invest in it, to be sure the AI search bots can read it.
- Run the live check when robots.txt allows every bot but AI still never cites your site.
- Look at the Competitors tab to see whether a competitor lets in AI bots that you block, or the other way round.