ChatGPT uses separate bots for training, searching, and taking action:
- GPTBot provides training data.
- OAI-SearchBot gathers data to respond to specific prompts.
- ChatGPT-User accesses pages when requested by users.
Knowing which bot is responsible for which task is essential before attempting to disallow it.
GPTBot
GPTBot locates information to build and update training data, ChatGPT’s knowledge base for providing answers.
ChatGPT doesn’t store training URLs or track where the info comes from. Disallowing this bot will prevent the platform from using your content for training, but it won’t impact your traffic. It may affect what ChatGPT understands about your company, though external sources likely provide that information, too.
Some publishers disallow the bot to prevent ChatGPT from learning from their content and to reduce costs, as AI bots can increase hosting needs and slow down servers, especially for large sites.
I typically suggest allowing access to GPTBot to provide first-hand information about a business and thus control the context.
ChatGPT updates training data regularly, usually with each release.
OAI-SearchBot
OAI-SearchBot searches the web for current information, user reviews, product details, and more.
Opinions differ as to whether the platform indexes the URLs from these searches. (ChatGPT states it “uses a hybrid system that includes limited indexing, plus on-demand retrieval, rather than a single, exhaustive web index.”)
OAI-SearchBot searches Google, Bing, Reddit, and others for info, much like humans, and may independently crawl sites, too.
Disallowing this bot prevents it from visiting your site, but it may still cite your pages via external links. Google does this, too, incidentally. A robots.txt file can prevent Google’s bot from crawling a site, but the search giant can still index and rank its pages.
Still, disallowing OAI-SearchBot will likely reduce or eliminate citations (and traffic), which is why I don’t usually advise it.
ChatGPT-User
ChatGPT-User performs actions as requested by users. For example, a user can prompt ChatGPT to visit a page and summarize its content.
ChatGPT-User does not provide training data or citations. If your server logs include this bot, a human instructed ChatGPT to visit your site. There’s no way to block this bot because it’s user-initiated, per ChatGPT.