Troubleshooting
Why does Crawlability say Could not read robots.txt?
Crawlability says Could not read robots.txt when your robots.txt could not be downloaded or was not a text file. Typical causes are a timeout, a server error, or a web page returned instead of the file. The page shows the reason.
What happens meanwhile
No bot gets a status until the file is readable. LiveMentions never shows Allowed as a guess.
What to do
- Open yoursite.com/robots.txt in a browser and check that it shows plain text.
- Fix the server answer, for example the error or the redirect to a web page.
- Press Reload robots.txt on the Crawlability page.
If you have no robots.txt
A file that is simply not found is different. It means no rules, so every bot is allowed, and the page says exactly that.
For more, see Crawlability.