robots.txt— definition

Conversely, if you write Disallow: /, you can completely shut these AI crawlers out.

GEO best practices

Allow everything by default is the iron rule of GEO. Setting Disallow makes you completely invisible in AI search.

Common mistakes

• Unintentional Disallow: many companies still disallow GPTBot via old templates (templates from before 2024 need review)
• No configuration at all: with no configuration, the default is Allow, so this is at least tolerable
• Disallowing AI bots only: the desire not to be used as training data is understandable, but you also lose citation opportunities

Related files

For GEO, robots.txt is typically maintained alongside:

• sitemap.xml: explicitly tells crawlers which URLs to visit
• llms.txt: conveys the main content to AI in Markdown

← Back to glossary top