In One Sentence
AI Crawler is the umbrella term for automated bots that crawl and fetch web pages on behalf of generative AI (ChatGPT / Claude / Gemini, etc.).
What does this look like in practice?
For example, when you publish a new blog post on your site, within hours to days OpenAI's "GPTBot" arrives and reads the article. The content it reads becomes part of ChatGPT's training data or a target of SearchGPT's real-time search.
Representative AI crawlers:
| Bot | Provider |
|---|