In One Sentence
AI Crawler is the umbrella term for automated bots that crawl and fetch web pages on behalf of generative AI (ChatGPT / Claude / Gemini, etc.).
What does this look like in practice?
For example, when you publish a new blog post on your site, within hours to days OpenAI's "GPTBot" arrives and reads the article. The content it reads becomes part of ChatGPT's training data or a target of SearchGPT's real-time search.
Representative AI crawlers:
| Bot | Provider | Purpose |
|---|---|---|
| GPTBot | OpenAI | ChatGPT / SearchGPT |
| Claude-Web | Anthropic | Claude's web search |
| Google-Extended | Gemini / AI Overview | |
| PerplexityBot | Perplexity | Perplexity search |
Why it matters
- If AI crawlers cannot reach you, you are not cited by AI: Effectively the same as not existing
- Controllable via robots.txt: Allow them and they come; Disallow them and they don't
- The most basic GEO tactic: It all starts with Allowing AI crawlers
How do you configure it?
Allow each crawler in robots.txt:
User-agent: GPTBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Google-Extended
Allow: /
For details, see robots.txt and the individual bot pages.