How AI search systems find and cite websites
AI answers in ChatGPT, Perplexity, Claude, Copilot, and Google draw on web pages that a crawler has already fetched. When someone asks a question, the system retrieves relevant pages and writes an answer with links to its sources. A page the crawler cannot reach or read is unlikely to be cited.
Each company runs separate crawlers for separate jobs. Blocking a training crawler does not remove you from search results. Blocking a search crawler can.
- Google: AI Overviews and AI Mode use pages from Google Search. A page must be indexed and eligible to show a snippet. Google-Extended controls Gemini training and grounding. Google says it does not affect inclusion or ranking in Search.
- OpenAI: OAI-SearchBot surfaces sites in ChatGPT search. GPTBot collects content that may be used to train models. ChatGPT-User visits pages when a user asks. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
- Perplexity: PerplexityBot surfaces and links sites in Perplexity search. Perplexity says it does not crawl content for AI foundation models. Perplexity-User fetches pages for user requests.
- Anthropic: Claude-SearchBot indexes pages to improve Claude's search results. ClaudeBot collects training data. Claude-User fetches pages when a Claude user asks a question.
- Microsoft: Bingbot crawls for Bing. Bing Webmaster Tools now reports when your pages are cited in Copilot and Bing AI answers.
