robots.txt and llms.txt for AI crawlers: a copy-paste setup
Short answer: if you want to appear in AI answers, do not block the search-type AI crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot) in robots.txt, list your sitemap, and publish a plain-text /llms.txt that says who you are, what you sell, your prices and your key URLs. Decide separately whether you also allow model-training crawlers (GPTBot, Google-Extended, CCBot).
The crawlers that matter
| User-agent | Run by | What it's for |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search results |
| GPTBot | OpenAI | Collecting content that may be used to train models |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to; robots.txt may not apply |
| ClaudeBot | Anthropic | Crawling for Claude |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Google-Extended | A control token for use in Gemini models; separate from Googlebot search | |
| Bingbot | Microsoft | Bing search, which also powers Copilot answers |
OpenAI documents its crawlers at developers.openai.com/api/docs/bots and notes robots.txt changes can take about a day to apply.
Copy-paste robots.txt (allow everything, list the sitemap)
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you don't want your content used for training, change the GPTBot and Google-Extended blocks to Disallow: / and keep the search crawlers allowed.
A minimal llms.txt
llms.txt is a proposed convention: a Markdown file at your site root that summarizes the site for language models. It is not a ranking factor anyone has confirmed, but it is cheap, and it gives answer engines one clean source of your facts.
# Example Studio
> Example Studio is a two-person bakery in Busan selling custom cakes from ₩45,000, ordered online with 3 days' notice.
## Key pages
- [Cakes and prices](https://www.example.com/cakes/): sizes, flavours, prices
- [How to order](https://www.example.com/order/): lead time, pickup and delivery
- [FAQ](https://www.example.com/faq/): allergens, cancellations
## Facts
- Location: Busan, South Korea
- Hours: Tue–Sun 10:00–19:00 KST
Add /llms-full.txt with the full text of your key pages if you want to give one file with everything.
Then check three more things
- Sitemap: every page you want found, with
lastmoddates, linked from robots.txt. - JSON-LD: Organization plus Service or Product with Offer, and FAQPage on your FAQ. Keep it identical to visible text.
- Your host or CDN: some firewalls block AI crawlers by default even when robots.txt allows them. Check your bot-protection settings.
What we saw in practice
In our 15 November 2026 check of 10 brand websites, only 3 named AI crawlers explicitly in robots.txt and 5 had an llms.txt with a real brand summary. This is one of the cheapest gaps to close. Our GEO audit includes a ready-to-paste robots.txt, llms.txt and JSON-LD for your site.
Frequently asked questions
Should I block GPTBot?
Only if you don't want your content used for model training. Blocking GPTBot does not stop OAI-SearchBot, which is the crawler for ChatGPT search results.
Is llms.txt required?
No. It is a proposed convention, not a confirmed ranking factor, but it is cheap to add and gives AI tools one clean summary of your business.
Where do robots.txt and llms.txt go?
Both go at the root of your domain, for example example.com/robots.txt and example.com/llms.txt.
Start: email yjs121256@gmail.com with your site (or product link), what you need and your deadline. You get a fixed written quote, usually within one Korean business day.