What happened
A recent Nginx probe revealed varied web content access methods among major AI assistants. ChatGPT, Claude, Perplexity, Meta AI, and Manus identify themselves with unique user-agents, performing live fetches from origin servers; ChatGPT-User, for instance, initiates multi-IP bursts for candidate pages. Conversely, Gemini, Copilot, and Grok do not announce themselves with distinct user-agents for live retrieval: Gemini answers entirely from its index, while Copilot and Grok appear as generic browser traffic, indistinguishable from human visitors.
Why it matters
This varied AI access mechanism complicates web analytics and content monetisation strategies. Platform engineers face challenges in accurately attributing traffic sources and implementing effective rate-limiting, as generic user-agents obscure AI-driven requests. For security architects, differentiating legitimate AI fetches from malicious bot activity becomes more complex without clear identification signals. This follows Google's recent efforts to combat SEO manipulation of AI search, where Gemini's reliance on the Googlebot user-agent creates a structural gap in vendor-specific traffic measurement and content management.
