Training crawlers
Scrape your content to train large language models.
GPTBotClaudeBotCCBot
Traffic back: none
AI search crawlers
Index your pages for AI-powered search results.
OAI-SearchBotPerplexityBot
Traffic back: some
Real-time fetchers
Grab pages on demand when a chatbot needs fresh data.
ChatGPT-User
Traffic back: occasional