Troubleshooting

Why does Crawlability say Could not read robots.txt?

Crawlability says Could not read robots.txt when your robots.txt could not be downloaded or was not a text file. Typical causes are a timeout, a server error, or a web page returned instead of the file. The page shows the reason.

What happens meanwhile

No bot gets a status until the file is readable. LiveMentions never shows Allowed as a guess.

What to do

  1. Open yoursite.com/robots.txt in a browser and check that it shows plain text.
  2. Fix the server answer, for example the error or the redirect to a web page.
  3. Press Reload robots.txt on the Crawlability page.

If you have no robots.txt

A file that is simply not found is different. It means no rules, so every bot is allowed, and the page says exactly that.

For more, see Crawlability.