Different AI engines find and read your docs in different ways. The behaviors overlap, but the differences matter if you care about specific surfaces.
How each gets to your docs
- ChatGPT (browse and search): uses a web search backend, then visits a small number of result pages. Treats your page the way a search engine would.
- Claude (web search): similar flow. Picks a few pages, cites them inline.
- Perplexity: crawls and indexes its own corpus. Heavier on citation. Often picks more sources per answer.
All three honor robots.txt for their search and crawl agents. The specific user-agent strings change; check each engine's docs for the current list.
What works across all three
A few things help every engine:
- A reachable, accurate sitemap.
- HTML that renders the answer in the document body. Client-side rendering is a problem if the page is empty without JavaScript.
- Stable URLs. Citations break when URLs change.
- Clear titles and
<h1>tags. The headline often becomes the citation text.
Where they differ
- ChatGPT and Claude lean on the linked page's body. Perplexity also uses the snippet shown in its own results.
- ChatGPT seems to weight authority signals (links from other sites) similarly to traditional search. Perplexity less so.
- Claude pays attention to dates in the page body. A visible
dateModifiedor a date in the headline helps.
What to test
Pick three or four questions a user might ask about your API. Ask each engine. Look at whether your docs appear, what gets cited, and which page the engine landed on.
The result usually points at one or two pages that need work, not a sitewide problem.
