Optimize

Training, search and user bots

Each AI bot on the Crawlability page has a type.

Training bots

Training bots collect pages to train AI models. Blocking a training bot is a legitimate choice, and LiveMentions never shows it as an error.

Search bots

Search bots find pages to answer questions. Blocking a search bot can keep an engine from using your pages in answers, and the page says so in careful words, for example "This can keep ChatGPT from using your pages".

User bots

User bots fetch a page because a person asked for it. Blocking a user bot can also keep an engine from reading your page when a person asks about it.

Vendor-confirmed and Reported

A bot labelled vendor-confirmed has its name, type and vendor taken from the vendor's own documentation. A bot marked Reported is listed only by third-party bot directories, so the page shows its type as Unconfirmed instead of guessing.

Should I block training bots?

That is a business choice. LiveMentions shows the status and does not call it an error. Search and user bots are the ones that affect whether AI can use your pages in answers.

For more, see Crawlability.