Optimize
Comparing your robots.txt with competitors
The Competitors tab on the Crawlability page puts your robots.txt next to the robots.txt of the competitors you track.
What you see
- A summary for each competitor, showing how many bots it leaves fully open.
- A bot-by-bot grid with You and the competitors you choose.
- A switch, Only show bots treated differently, that hides every bot where you and the chosen competitors match.
What is read
Only each competitor's public robots.txt is read. The statuses mean what that file says about the homepage and its rules. No sitemap is checked for competitors, and the page says so. A competitor whose file cannot be read shows Could not read, never a status.
If the tab is empty
The tab compares the competitors you track. With none tracked, it asks you to track some on the Competitors page.
For more, see Crawlability: the URL tester and competitors.