OpenAI crawlers
This directory covers 3 bots that OpenAI runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against OpenAI's documentation on .
| Bot | Type | What it does | robots.txt token |
|---|---|---|---|
| ChatGPT-User | AI user-triggered fetchers | Visits a page when a ChatGPT or Custom GPT user's request needs it. | ChatGPT-User |
| GPTBot | AI training and dataset crawlers | Crawls content that may be used to train OpenAI's foundation models. | GPTBot |
| OAI-SearchBot | AI search crawlers | Surfaces websites in ChatGPT's search features. | OAI-SearchBot |
How to verify a request from OpenAI
A user agent is a string any client can send, so the name alone proves nothing. Check an address from your logs, or apply the operator's own method:
- ChatGPT-User
- Check that the source IP is in the ranges OpenAI publishes for ChatGPT-User.https://openai.com/chatgpt-user.json
- GPTBot
- Check that the source IP is in the ranges OpenAI publishes for GPTBot.https://openai.com/gptbot.json
- OAI-SearchBot
- Check that the source IP is in the ranges OpenAI publishes for OAI-SearchBot.https://openai.com/searchbot.json