AI crawler user-agent list 2026: 14 bots and robots.txt tokens to know

TL;DR

An AI crawler user agent identifies the bot requesting a page and often reveals whether it is collecting training data, indexing content for AI search, or fetching a URL for a user. In 2026, visibility-focused sites should allow search and user-fetch agents, then make a separate policy decision about training crawlers.

What is an AI crawler user agent?

An AI crawler user agent is a name sent with an automated web request that identifies the company, bot, and sometimes the request's purpose. Site owners use that name in server logs, robots.txt, CDN rules, and web application firewalls to decide which automated systems can access their content.

The phrase "AI crawler" hides several different jobs. Some bots collect training data. Others build search indexes. A third group fetches one page because a person asked an AI assistant a question.

There is also a fourth category that is easy to miss. Google-Extended and Applebot-Extended are control tokens, not separate crawlers. You can place them in robots.txt, but you should not expect either name to appear as a standalone request in your server logs.

For a broader introduction to these roles, read AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and How to Let Them In.

What is the AI crawler list for 2026?

The 2026 visibility-first list contains 14 relevant user-agent tokens across six AI companies. Nine directly affect search, answer generation, or user-requested fetches. Three primarily collect training data. Two, Google-Extended and Applebot-Extended, control permitted uses of content gathered by another crawler.

These roles come from the companies' own crawler documentation: OpenAI, Anthropic, Perplexity, Google, Meta, and Apple.

This list focuses on bots with a documented connection to prominent AI answer products. It does not include every commercial scraper, social-preview crawler, SEO bot, or undisclosed agent that may visit a website.

What is the difference between GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot?

GPTBot and ClaudeBot primarily collect possible training data. OAI-SearchBot and Claude-SearchBot support AI search visibility. PerplexityBot indexes and retrieves pages for cited answers rather than training foundation models. Treating all four as interchangeable can remove a site from AI answers when the owner only intended to reject training use.

OpenAI explicitly lets publishers allow OAI-SearchBot while disallowing GPTBot. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, although navigational links may still appear.

For more detail on OpenAI's three-agent system, read GPTBot Explained: How ChatGPT Crawls, Sees, and Cites Your Site.

Anthropic provides a similar separation. ClaudeBot, Claude-SearchBot, and Claude-User have independent roles and must be addressed separately in robots.txt.

Perplexity makes a sharper distinction. Its documentation says PerplexityBot supports search results and does not collect training data. Perplexity-User retrieves individual pages after user requests and generally does not follow robots.txt.

Which AI crawlers should I allow?

Sites that want AI citations should allow the nine search and user-fetch tokens. Decide separately whether to allow the three training crawlers and two usage-control tokens. This keeps pages eligible for current answers without automatically treating every model-training use as a requirement for visibility.

Google is the awkward exception. Google says Google-Extended controls both training for future Gemini models and some grounding in Gemini products. It does not affect Google Search inclusion or rankings. Publishers cannot assume it is a clean training-only switch.

Apple separates these uses more cleanly. Blocking Applebot-Extended prevents training use while leaving pages eligible for Apple search products. The separate nosnippet directive controls whether Applebot can use a page as current context in an AI-generated answer.

What robots.txt rules should I use for AI crawlers?

Use a separate robots.txt group for every agent whose behavior you want to control. Exact groups make the policy readable and prevent one training decision from accidentally blocking a search agent. Remember that robots.txt expresses crawl preferences. It does not override a CDN, firewall, login wall, or rate limit.

How do I block AI training but allow AI search?

Allow each search crawler and user fetcher, then disallow the corresponding training crawler. The configuration below preserves visibility across ChatGPT, Claude, Perplexity, Meta AI, Google Search, and Apple products. Google remains an exception because Google-Extended combines Gemini training and grounding controls.

# OpenAI search and user retrieval
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# OpenAI training
User-agent: GPTBot
Disallow: /

# Anthropic search and user retrieval
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Anthropic training
User-agent: ClaudeBot
Disallow: /

# Perplexity search and user retrieval
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Google Search and AI Overviews
User-agent: Googlebot
Allow: /

# Google combines Gemini training and grounding in this token.
# Keep Allow for Gemini grounding, or change to Disallow to reject both.
User-agent: Google-Extended
Allow: /

# Meta user retrieval
User-agent: meta-externalfetcher
Allow: /

# Meta training
User-agent: meta-externalagent
Disallow: /

# Apple search, Siri, and AI answers
User-agent: Applebot
Allow: /

# Apple model training
User-agent: Applebot-Extended
Disallow: /

OpenAI and Perplexity say crawler-policy changes may take about 24 hours to propagate. A successful update does not force an immediate recrawl or citation.

The Robots Exclusion Protocol also applies per host. If your content lives at docs.example.com, rules published only at example.com/robots.txt do not control the documentation subdomain.

Why do AI crawler controls matter in 2026?

AI requests are growing faster than human web traffic and place more load on origin servers. They also have different commercial value. A training crawler may consume thousands of pages without producing a visit, while a user fetcher may represent someone actively researching a product, vendor, or purchase.

Fastly measured AI requests across a fixed customer cohort from January through May 2026. Its findings included:

• AI request volume grew about 30 percent, roughly 6.5 times faster than human traffic.

• Claude-related traffic increased more than 555 percent from its January baseline.

• Crawlers produced 85 percent of AI bot requests in May, while user-linked fetchers produced 15 percent.

• More than 51 percent of AI requests reached origin servers, compared with less than 9 percent of human requests.

Those numbers explain why "allow every bot" and "block every bot" are both crude policies. Site owners need to distinguish identity and intent.

"AI traffic is fundamentally changing how the internet operates."

Artur Bergman, founder and CTO at Fastly, in Fastly's June 2026 research announcement

How do I verify AI crawler traffic?

Verify AI crawler traffic in server, CDN, or WAF logs. Search for the documented token, inspect the requested URL and response status, then validate the source using the vendor's published IP ranges or DNS procedure. A user-agent match by itself proves nothing because any client can copy that string.

A basic log search might look like this:

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|meta-externalagent|meta-externalfetcher|Applebot' access.log

For every match, record:

1. The user-agent token.

2. The requested page.

3. The response status.

4. The source IP.

5. Whether the response contained the complete main content.

6. Whether the request reached cache or origin.

Healthy requests to public content usually return 200. Repeated 403 responses point to a firewall or access rule. A run of 429 responses means rate limiting is interfering. 500 responses suggest the crawler is exposing an application or origin failure.

Do not search logs for Google-Extended or Applebot-Extended. They are policy tokens used in robots.txt, not separate HTTP user agents.

OpenAI and Perplexity publish current IP data for their crawlers. Anthropic publishes a source-IP list. Google documents both published IP ranges and reverse-DNS verification. Use those live sources instead of copying a permanent IP allowlist into an article or configuration file.

Why is an allowed AI crawler still getting blocked?

An Allow rule only removes a robots.txt restriction. AI crawlers can still fail at the CDN, WAF, application, rendering, or authentication layer. A site may look open in its robots file while returning a challenge page, empty client-rendered shell, login screen, or 403 response to the actual bot.

The test that matters is not whether robots.txt looks correct. The test is whether a verified crawler receives the canonical page, complete content, and a usable 200 response.

Does allowing AI crawlers make my brand appear in AI answers?

Allowing AI crawlers makes a page eligible to be found or fetched. It does not make the page worth citing. AI systems still choose among accessible sources based on relevance, authority, freshness, evidence, and how directly each content block answers the user's question.

Crawler access is the first layer:

1. Can the engine reach the page?

2. Can it read the main content?

3. Does the page answer a specific prompt?

4. Are its claims supported?

5. Does the engine cite it instead of another source?

Server logs answer the first two questions. They cannot tell you whether ChatGPT, Claude, Perplexity, or Gemini used the page in an answer.

Anagram measures that next layer. It tracks the prompts a brand cares about, which sources the engines cite, and which competitors receive the citation when the brand does not. That separates an access problem from a content problem.

A crawler visit is evidence of access. A citation is evidence that the content won.

What else should site owners know about AI crawler controls?

AI crawler policies need regular review because vendors add agents, split existing roles, and change verification methods. Keep the policy in version control, document why each token is allowed or blocked, and compare the file against current vendor documentation whenever a new crawler appears in your logs.

Does allowing GPTBot make my site appear in ChatGPT Search?

No. GPTBot is OpenAI's training crawler. OAI-SearchBot controls eligibility for ChatGPT search answers, while ChatGPT-User handles certain user-triggered visits. Allow OAI-SearchBot if ChatGPT visibility is the goal, even if your policy blocks GPTBot.

Can I allow Claude search while blocking Claude training?

Yes. Allow Claude-SearchBot and Claude-User, then disallow ClaudeBot. Anthropic documents all three as independent agents that respect robots.txt, so the training decision does not have to remove your pages from Claude's current search and user-retrieval systems.

Does blocking Google-Extended hurt Google rankings?

No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. It controls whether Google-crawled content may support future Gemini training and certain grounding uses. Continue allowing Googlebot if you want visibility in Google Search and its AI search features.

Can user-triggered AI fetchers ignore robots.txt?

Some can. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it because the request begins with a user action. Enforce a hard allow or block at the CDN, WAF, or application layer using the vendor's verification data.

Should I use one wildcard rule for every AI crawler?

Use explicit groups when different bots receive different permissions. A wildcard is fine when every crawler should follow the same policy, but it cannot express the distinction between training, search indexing, and user retrieval. Explicit groups also make audits and future changes safer.

Which sources support this AI crawler list?

This guide uses first-party crawler documentation, the formal robots.txt specification, and Fastly's measured network data. Crawler names and policies change, so vendor pages remain the source of truth. Check them again before publishing a production access policy or permanent WAF rule.

1. OpenAI, Overview of OpenAI Crawlers

2. Anthropic, Does Anthropic crawl data from the web?

3. Perplexity, Perplexity Crawlers

4. Google, Google's common crawlers

5. Meta, Web crawlers

6. Apple, About Applebot

7. IETF, RFC 9309: Robots Exclusion Protocol

8. Fastly, AI Traffic Grew 6.5x Faster Than Human Traffic This Yeart’