The URL tester is the second tab on the Crawlability page. It checks one exact address on your website for every AI bot.

How to use it

1. Open the URL tester tab.
2. Type an address on your own website. Addresses on other websites are refused.
3. Read the result for each bot: allowed or blocked, and the rule line that decides.

The live check

Switch on Also check how my site answers these bots to make one request per key bot, using that bot's user agent, and see the real server answer, for example 200 or 403. This catches a firewall or CDN that answers AI bots differently from a browser even when robots.txt allows them. The row says Different from a browser when that happens.

The live check is a simulation. Some firewalls check where a bot comes from, so a real crawler may be treated differently.

Who can use it

The URL tester needs an editor role, because it makes real requests to your website.

For more, see Crawlability: the URL tester and competitors.