The AI Crawler Guide for Local Businesses (and What Your Robots.txt Should Actually Say)
Anthony (Tony) Velte
Founder & Principal · Author of 12+ books
AI crawler access is a plumbing problem with a short list of correct answers. Major AI operators now separate their jobs: one identity for collecting training data, another for building a search index, another for fetching a page live when a person asks for it. Some of those identities are crawlers with their own user agents, and some, at Google and Apple, are robots.txt control tokens rather than separate fetchers. The jobs are controlled independently, so a decision about one of them says very little about the others. This guide lists the agents each operator documents, explains what each one is for, and then covers the part most copy-paste robots.txt allowlists get wrong: a per-bot group written carelessly can hand those bots the private paths you meant to keep closed.
Blocking a Training Crawler and Being Invisible Are Two Different Things
In late 2023 a wave of advice told website owners to block GPTBot so that OpenAI could not train on their content. A lot of small businesses copied that rule. Two things about it are worth stating precisely. First, GPTBot is the training crawler. OpenAI documents a different agent, OAI-SearchBot, for ChatGPT Search indexing, and a third, ChatGPT-User, for fetches that a person triggers inside a conversation. Disallowing GPTBot opts you out of training-data collection and leaves the other two agents untouched. Second, a rule written today has no effect on models that were already trained.
For a publisher licensing an article archive, the trade-offs run in several directions at once. For a local business whose website exists so that customers can find it, there is usually little reason to close any of these doors. That is a judgment about your own business model, though, and worth treating as one. No AI operator publishes a ranking benefit for allowing its crawler.
AI crawler access is one of the six dimensions we check in SignalScore, our working methodology for scoring how reachable a business is to AI engines. We weight it at 10%. That weight is our own judgment about where effort pays off, and it is not a published property of any AI platform. It sits high on our list for a simple reason: it is cheap to fix, and nothing else we recommend does any work if the crawler cannot fetch the page.
Four Jobs, Four Kinds of Agent
Operators have split their crawling into distinct, separately controllable jobs. Once you can place an agent in the right group, what a rule about it will and will not do becomes obvious.
The four functions, and which agents belong to each:
- Training and model development. These crawlers collect publicly available pages that may be used to train or improve a model. GPTBot, ClaudeBot, MistralAI-Training, and meta-externalagent sit here.
- Search indexing. These crawlers build the index that an AI search product queries when it composes an answer. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, and meta-webindexer sit here.
- User-directed retrieval. When somebody pastes a URL or asks a question that sends the assistant to fetch a page right then, a separate agent does the fetching. ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User, and meta-externalfetcher sit here.
- Control tokens. Google-Extended and Applebot-Extended have no user-agent string and fetch nothing at all. They exist so that a site can tell Google and Apple how data their ordinary crawlers already collected may be used.
OpenAI
OpenAI documents three agents and states that each is controlled independently of the others.
OpenAI crawlers:
- GPTBot is the training crawler. It collects publicly available content for training and improving OpenAI models. Disallowing it opts you out of that collection and does nothing else.
- OAI-SearchBot indexes pages for ChatGPT Search. This is the agent that governs whether your pages are eligible to surface there, and it is a separate decision from the GPTBot one.
- ChatGPT-User fetches a page in real time on behalf of somebody in an active conversation, for instance when a question they ask leads ChatGPT to visit a specific page. OpenAI notes that robots.txt may not apply to these user-initiated fetches, since the request originates with a person.
Anthropic
Anthropic currently documents three agents, split along the same lines.
Anthropic crawlers:
- ClaudeBot collects content for training and model development.
- Claude-SearchBot indexes pages for search-grounded answers.
- Claude-User fetches a page when a person in a conversation directs Claude to it.
Two older names still circulate widely in published allowlists: Claude-Web and anthropic-ai. Neither appears in Anthropic current crawler documentation. A robots.txt rule that mentions them does no harm, and it also does no work, so treat them as legacy rather than as live agents to plan around.
Perplexity
Perplexity crawlers:
- PerplexityBot is the search indexing crawler. Perplexity states that the content it collects is used for search rather than for foundation-model training. Perplexity answers cite their sources by design, which makes this a consequential surface for referral traffic to a local business.
- Perplexity-User handles fetches that a person requests. Perplexity documents that this agent generally ignores robots.txt, on the reasoning that the request came from a user rather than from an automated crawl. A robots.txt rule is therefore not a reliable control for it in either direction.
Googlebot is the crawler. It builds the Search index, and Google documents that Googlebot renders JavaScript. Google AI features, including AI Overviews, are grounded in that same Search index, so ordinary Search crawlability is what governs eligibility. There is no separate AI-Overviews crawler to allow.
Google-Extended is where most write-ups go wrong. It is a robots.txt control token: it has no user-agent string and it fetches nothing at all. It governs whether content that Google already collected may be used for Gemini training and grounding. Google documents that a Google-Extended rule has no effect on Search inclusion, on ranking, or on AI Overviews. Google published guidance on AI-search optimization also states that no special markup is required for generative results and that Google ignores llms.txt files.
Apple
Apple agents:
- Applebot is the crawler behind Siri, Spotlight, and Apple search features.
- Applebot-Extended is a control token in the same mold as Google-Extended. It does not crawl anything. It governs whether data that Applebot already fetched may be used to train Apple foundation models.
Amazon
Amazon has split what used to be one agent into three purpose-separated ones, which makes older allowlists that name only Amazonbot incomplete.
Amazon crawlers:
- Amazonbot collects content for product improvement and training.
- Amzn-SearchBot handles search indexing, which is the path that governs Alexa answer eligibility.
- Amzn-User handles live requests that a person triggers.
Mistral
Mistral documents three agents with the split stated in the names themselves: MistralAI-Training for training data, MistralAI-Index for search indexing, and MistralAI-User for fetches during query handling. Older lists that carry only MistralAI-User are describing one third of the picture.
Meta
Meta documents meta-externalagent for training, meta-webindexer for search indexing, and meta-externalfetcher for user-directed fetches. Meta AI is built into WhatsApp, Instagram, and Facebook, which makes this a large distribution surface for a consumer-facing local business.
Common Crawl
CCBot collects pages for Common Crawl, an open web dataset published for anyone to use. Many AI labs have trained on Common Crawl data, and the published GPT-3 paper lists it as the single largest source in that model training mix. One decision about CCBot therefore sits upstream of several model ecosystems at once.
Bytespider
Bytespider is attributed to ByteDance. It is the one agent in this guide with no first-party documentation to point at: no published statement of purpose, no IP ranges to verify requests against, and no stated robots.txt policy. Site operators and CDN vendors have widely reported crawling from it that does not match the directives in robots.txt. Treat its behavior as unverified, and if its request volume becomes a problem, handle it at the CDN or web-server layer, where the control is actually enforced.
Names in Circulation That Do Not Hold Up
Copy-paste allowlists travel faster than operator documentation changes, so a handful of entries recur across published lists with no current source behind them. Three come up constantly.
Entries to drop from an inherited allowlist:
- cohere-ai. Cohere currently documents no web crawlers or user agents at all. A rule naming cohere-ai matches nothing.
- CopilotBot. Microsoft authoritative crawler list does not include it. Copilot grounding rides on bingbot, so bingbot is the agent to reason about for Microsoft surfaces.
- Claude-Web and anthropic-ai. Retired Anthropic names, superseded by the three current ones listed above.
None of these rules cause damage by themselves. They cost credibility when a list is presented as verified, and they inflate the per-bot group count in a robots.txt file, which leads directly to the structural mistake in the next section.
What Your Robots.txt Should Actually Say
Robots.txt behavior is specified in RFC 9309, and two rules in that specification decide almost everything about a sound AI crawler configuration.
The two rules that matter:
- Allowed is the default. A URL with no matching Disallow rule is allowed. An explicit "Allow: /" adds nothing unless it is overriding a more specific Disallow rule.
- Every group that names a crawler's own user agent is combined and applied to it; the wildcard group is used only when no group names that crawler at all. So once any group names a bot directly, the wildcard group stops applying to that bot entirely.
Put those two together and the standard AI allowlist pattern turns into a hole in your site. Take a file with a wildcard group that disallows /admin/, /api/, and /thank-you/, followed by twenty per-bot groups that each contain only "Allow: /". That file has just told twenty AI crawlers they may fetch /admin/ and /api/. The wildcard disallows never reach them, because those bots matched a group of their own. The site owner added per-bot groups to be more welcoming and removed their private-path protection for the exact agents they were writing rules for.
For most sites the correct configuration involves no per-bot groups at all. If your wildcard group disallows only a handful of private paths, every compliant AI crawler in this guide already has access to everything else by default, with no further work. Write a per-bot group only when you want that bot to have different rules from everyone else, and repeat your disallows inside it when you do.
Here is a small, correct file. The wildcard group carries the real policy. The second group exists only because GPTBot is being given one extra exclusion, and it repeats the base disallows so that adding it does not open anything up.
# Applies to every crawler that has no group of its own.
# Everything not listed here is allowed by default (RFC 9309).
User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /thank-you/
# Only needed because this bot gets DIFFERENT rules from the wildcard group.
# Note that the three base disallows are repeated. Without them, this group
# would grant GPTBot access to /api/, /admin/, and /thank-you/.
User-agent: GPTBot
Disallow: /api/
Disallow: /admin/
Disallow: /thank-you/
Disallow: /internal-pricing/
Sitemap: https://your-domain.com/sitemap.xmlIn that file every AI crawler named in this guide can reach every page except the three private paths, because allowed is the default and no rule says otherwise. GPTBot carries one additional exclusion. Delete the second group and the configuration is still correct for every bot; you lose only the GPTBot-specific rule.
In a Next.js App Router project, the cleanest way to ship the file is the robots.ts metadata route, which generates it at build time from a typed configuration. That is the approach we use on our own site, and the live file is served at localstardigital.com/robots.txt.
How to Verify Crawler Access for Real
Robots.txt is advisory. It expresses a preference that compliant crawlers honor, and it enforces nothing at the network layer. Verification therefore means checking two separate questions: whether your file says what you intended, and whether real crawler traffic matches it.
Three checks, in the order they are worth running:
- Read the file you actually serve. Fetch https://your-domain.com/robots.txt and read it top to bottom, checking group order and whether every specific group repeats the disallows it needs. Setting a fake user-agent header on that request changes nothing about the response. It returns the same static file either way and tells you nothing about how any crawler behaves.
- Review server or CDN access logs for the documented agents. Filter for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, and whichever others matter to you, then look at which paths they requested and what status codes came back. Where an operator publishes IP ranges, validate that requests claiming an agent name actually originated inside them. User-agent strings are trivially spoofed, and a meaningful share of traffic claiming to be a major AI crawler is something else.
- Check the bot controls in your CDN or WAF. Cloudflare and comparable providers ship AI-crawler blocking that is enabled by default on some plans, and those controls are enforced at the edge regardless of what robots.txt says. A site can serve a perfect robots.txt file and still return 403 to every AI crawler because of a setting nobody remembers turning on. In our audits this is the most common cause of unexplained blocking, and it is invisible from the robots.txt file alone.
There is no schedule for when a crawler will next visit. A rule that permits crawling grants permission; it summons nobody. Crawl frequency is the operator decision, driven by their own discovery and scheduling, so treat log review as an ongoing check with no deadline attached.
Crawler access is one of six dimensions in SignalScore, our working methodology for scoring how visible a business is to AI engines. For how the page-level signals fit together once the crawler can reach you, read our 10-point AI visibility checklist. For the scoring model itself, see our SignalScore page, and for the audit-and-fix engagement built around it, see our GEO services.
The robots.txt problems we find are usually structural: a per-bot group that drops the wildcard disallows, or a CDN bot setting nobody remembers enabling. A missing bot name is rarely the issue. Send us your domain and we will read your live robots.txt and check your CDN bot controls against the current operator documentation before you change anything.
Frequently Asked Questions
No. GPTBot is OpenAI training crawler. ChatGPT Search inclusion is governed by a different agent, OAI-SearchBot, and live fetches during a conversation come from a third, ChatGPT-User. Disallowing GPTBot opts you out of training-data collection and leaves the other two alone. It also has no effect on models that were already trained, since the content those were built from was collected before your rule existed.
Almost certainly not. Under RFC 9309 a URL with no matching Disallow rule is allowed, so if your wildcard group excludes only a few private paths, every compliant crawler already has access to everything else. Per-bot groups exist to give one bot different rules from the rest. When you write one, repeat your disallows inside it, because the wildcard group stops applying to any bot that has a group of its own. A per-bot group containing only "Allow: /" is the common version of this mistake, and it opens /admin/ and /api/ to that bot.
Allowing a training crawler does mean your publicly available pages may be collected for training future models. Whether that trade is worth making depends on what your website is for. A publisher licensing an archive and a local services business whose site exists to be found are in genuinely different positions, and the widely shared advice to block was written for the first case. Robots.txt is how the preference gets expressed either way, and because the major operators now document which agent handles which job, the choice can be made per job instead of as one blanket rule.
Server or CDN access logs, filtered for the documented user-agent strings and cross-checked against the operators published IP ranges where those exist. That is the only place the answer lives. Fetching your own robots.txt with a spoofed user-agent header tells you what your file says and nothing about crawler behavior. Check your CDN or WAF bot-management settings at the same time, since edge-level AI crawler blocking overrides robots.txt entirely and is enforced where robots.txt is only advisory.
For most local-business websites, no. If a specific crawler is generating problematic load, rate-limit it at the CDN or web-server layer, where the control is actually enforced. Bytespider is the usual candidate, and it is also the one agent here with no published documentation or IP ranges to verify against, which makes edge-level controls the practical tool for it rather than a robots.txt line.
Ready to improve your AI visibility?
Book a strategy call. We will audit your search and AI presence and recommend a plan tailored to your business.