Technical Health for AI Visibility: The Checks We Run and Why
Anthony (Tony) Velte
Founder & Principal · Author of 12+ books
Technical health for AI visibility means the pages you care about are fast, reliable, and readable by an automated client: the content is present in the HTML your server returns, the response arrives promptly, and nothing in your robots.txt or your CDN configuration blocks the agents you want to reach. It carries 15% of the weight in our SignalScore methodology, and it is the dimension that gates the others. A client that cannot fetch and parse a page has nothing to work with, whatever the quality of the writing on it.
Technical Health is not the most heavily weighted dimension in our model. Citability and Brand Authority both count for more in our scoring, because they bear more directly on whether an engine has any reason to cite you. Technical Health is the prerequisite. It decides whether the rest of the work can be read at all, which is why we check it first.
Crawlers, Search Indexers, and Control Tokens Do Different Jobs
For most of the last decade, technical SEO meant optimizing for one crawler, and that crawler got steadily more capable. Googlebot executes JavaScript, renders pages in a Chromium-based environment, and waits for client-side hydration. Google Search Central still recommends server-side rendering or static pre-rendering for reliability, but it documents JavaScript-heavy sites as indexable (see Google Search Central's JavaScript SEO basics). Google's AI features, AI Overviews included, are grounded in that same Search index.
The AI operators look different, and the detail that most guidance skips is that they no longer run one bot for everything. Most major operators now publish separate identities for separate jobs, each controlled independently in robots.txt; at Google and Apple the training decision rides on a control token applied to their existing crawlers rather than a separate fetching agent:
What the operator documentation actually says about the agents people usually lump together:
- OpenAI runs GPTBot for training and model development, OAI-SearchBot for the indexing that feeds ChatGPT Search, and ChatGPT-User for fetches triggered when a person asks ChatGPT to look at a specific page. OpenAI notes that robots.txt may not apply to that last one, because the request originates with a user.
- Anthropic documents three agents: ClaudeBot for training and model development, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval. Claude-Web and anthropic-ai are legacy names that no longer appear in Anthropic's current documentation.
- Perplexity runs PerplexityBot for search indexing, which its documentation states is not used for foundation-model training, and Perplexity-User for fetches a person asked for. Perplexity states that the user-triggered agent generally ignores robots.txt for that reason.
- Google-Extended is not a crawler. It has no user-agent string and fetches nothing. It is a robots.txt control token governing whether content Google's existing crawlers already fetched may be used for Gemini training and grounding. Disallowing it has no effect on Google Search inclusion, ranking, or AI Overviews.
- CCBot belongs to Common Crawl, which publishes an open web corpus that many AI labs train on. It is not tied to any single engine's live answers.
That separation matters for decisions people make casually. Blocking GPTBot opts your site out of OpenAI training, and it does nothing at all to your presence in ChatGPT Search, which is governed by OAI-SearchBot. Treating "blocked from training" and "not citable" as the same condition produces the wrong robots.txt in both directions. Our AI crawler allowlist post works through the full agent list and the robots.txt syntax.
What Is Actually Known About How These Agents Read Pages
Operators publish which agents they run and what those agents are for. They do not publish rendering behavior, timeout windows, or render budgets. Anyone quoting you a figure for how long GPTBot waits before giving up is quoting a number nobody has released.
What remains is an engineering judgment, and the conservative one is this: assume an automated client reads the initial HTML response and nothing more. That assumption costs you nothing if a given agent turns out to render JavaScript, and it protects you completely if it does not. Server-rendered or statically generated content is the robust choice here for the same reason it is the robust choice for a slow phone on a bad connection. Googlebot is the documented exception, since it does render JavaScript, and Google's AI surfaces read the index Googlebot builds.
The Seven Technical Checks We Run
The list below is our checklist. It is not a published admission test, and no operator documents a rubric that a page passes or fails. What these seven items share is that each one makes a page easier for any automated client to fetch and read reliably, and each one also serves the humans who arrive on it. We work through them in this order because the earlier items block the later ones:
- HTTPS with a valid certificate. Browsers mark plain HTTP as not secure, mixed-content warnings break pages, and every extra redirect hop is another failure point. This is baseline web hygiene rather than an AI-specific requirement.
- Main content present in the server response. Server-side rendering and static site generation both satisfy this. It is the one item on this list we would keep if we could keep only one.
- A fast, stable first response. web.dev suggests keeping Time to First Byte under roughly 800 milliseconds as a guide for reaching a good Largest Contentful Paint, and we use it the same way: a target for real users, and a proxy for a server that will still answer when several clients hit it at once. Slow shared hosting and unoptimized backend queries are the usual causes.
- Core Web Vitals in the good range. These are Google's real-user experience metrics and they feed Google Search page experience signals. AI crawlers do not run Core Web Vitals audits. We check them because a site with poor vitals nearly always has other technical problems, and because the person who clicks through from an AI answer still has to use the page.
- Valid, parseable HTML. Unclosed elements and malformed markup make content extraction less predictable for everything that reads a page programmatically, from readability parsers to screen readers. No operator publishes a strictness penalty. Clean markup is simply cheaper to consume.
- A robots.txt that permits the agents you want. Under RFC 9309 the default is allowed, so a path with no matching disallow rule is already open and an explicit "Allow: /" is unnecessary. The real trap runs the other way: once a bot matches its own named group, the wildcard group stops applying to it entirely, so a per-bot group containing only "Allow: /" silently discards every disallow you wrote for /admin/ or /api/. Most sites need no per-bot AI groups at all.
- No JavaScript-only critical content. Headlines, body copy, prices, service areas, and contact details all belong in the served HTML. This is the failure we find most often in audits.
Verification matters as much as the checklist. Running curl with a spoofed GPTBot user agent against your own robots.txt proves nothing about crawler access; it fetches a text file and reports what that file says, which you could have read directly. Real verification is server or CDN log review for the documented user agents, validated against the IP ranges the operators publish, plus a look at your CDN bot-management settings. Cloudflare's AI crawler controls are enforced at the edge. robots.txt is advisory and depends on each agent choosing to honor it.
Core Web Vitals: What Each One Measures and What to Hit
Core Web Vitals are Google's standardized metrics for real-world page experience, measured from actual visits rather than from a crawler fetch. The current metrics and thresholds are documented at web.dev/vitals, maintained by Google's Chrome team. Three of them carry the weight.
Largest Contentful Paint (LCP): under 2.5 seconds
LCP measures how long the largest visible content element takes to render, usually a hero image, a headline block, or the main content area. Under 2.5 seconds is "good," 2.5 to 4 seconds is "needs improvement," and over 4 seconds is "poor." It is the closest single number to what a visitor experiences as page speed. Improving it generally means fixing server response time, render-blocking resources, or oversized images, and all three of those make the page easier for automated clients to read as a side effect.
Cumulative Layout Shift (CLS): under 0.1
CLS measures how much the layout shifts unexpectedly during load, the experience of a button moving as you reach for it or content jumping down because an ad finally arrived. Under 0.1 is "good." High CLS usually traces back to missing image dimensions, late-loading fonts, or content injected after first paint. It is an engineering discipline problem more than a performance problem, and it is usually cheap to fix once someone looks.
Interaction to Next Paint (INP): under 200 milliseconds
INP replaced First Input Delay as a Core Web Vital in March 2024 and measures how quickly the page responds to taps, clicks, and keypresses. Under 200 milliseconds is "good," 200 to 500 milliseconds is "needs improvement," and over 500 milliseconds is "poor." INP has no bearing on how any crawler reads your HTML. It has considerable bearing on whether the person who arrived from an AI answer stays on the page long enough to call you.
These thresholds come from Google's Chrome team and are the same ones Google Search uses for page experience. There is no separate AI crawler standard for Core Web Vitals, because AI crawlers do not measure them. We include the vitals in our technical checks as a site-quality standard, not as an admission test for AI engines.
The JavaScript-Only Trap
The most common technical failure we find in audit work is a client-side-only React, Vue, or Angular application where the main content does not exist in the served HTML. View source on one of those sites and you get a near-empty document with a single `<div id="root">` and a script tag. The headlines, body copy, product information, and contact details all get written into that div after the JavaScript bundle downloads, parses, and executes.
Browsers handle this. Googlebot handles it, because it renders JavaScript. What no operator will tell you is whether the other agents do, which leaves you betting your visibility on undocumented behavior. The conservative read is that a page whose content exists only after script execution is a fragile source for anything that is not a full browser, and that category includes text-extraction pipelines you will never hear about.
The fix is structural. Server-render the site (Next.js, Remix, Nuxt, and SvelteKit all support it), statically generate it at build time, or pre-render it through a service that emits real HTML. Middleware that sniffs user agents and serves different markup to bots is a maintenance liability and a cloaking risk, so we do not recommend it.
Our perspective on this comes from decades of enterprise systems work: if the foundation is wrong, polish on top does not fix it. AI visibility follows the same rule. Content that exists only after script execution is content you are trusting undocumented behavior to find. That is why we treat the technical layer as the gating dimension rather than the finishing touch.
How LocalStar's Own Technical Foundation Works
We run our own site, and every custom-build client site, on Next.js 15 with server-side rendering by default, deployed to Vercel's edge network. Every page is either pre-rendered at build time or rendered on the server per request, so a client fetching <code>localstardigital.com</code> receives populated HTML in the first response. We track Core Web Vitals continuously and treat a regression as a build-breaking issue rather than a "fix it later" item.
When we ran localstardigital.com through our own methodology, the technical dimension scored 100/100 on our quick diagnostic and 88/100 on our full GEO audit. Those are SignalScore numbers, produced by our own scoring model against our own checklist. They are an internal measure of how we do against the standard we set, and they are not independent validation from anyone. We published the full results, including the dimensions where we scored ourselves poorly. It is the same architectural standard we apply to the custom-build websites we deliver for Strategic-tier clients. See SignalScore methodology for the full dimension weightings and /services/web-design for how the architecture shows up in client builds.
Technical Health is the 15% you cannot skip. Fix the server response, get your content into the served HTML, keep your robots.txt honest about which agents you actually want, then verify it in your logs instead of assuming. The other dimensions of a SignalScore, Citability, Content Quality, Schema, AI Crawler Access, and Brand Authority, all rest on a page that can be fetched and read.
Most sites miss at least one of the seven checks and have no way to notice. Send us the URL and we will run it through all seven, then tell you which findings are a same-day fix and which need a rebuild.
Frequently Asked Questions
No operator publishes rendering guarantees for its crawlers, so nobody outside those companies can answer that with certainty. Googlebot is the documented exception: it renders JavaScript, and Google's AI features read the index Googlebot builds. For everything else, the safe engineering assumption is that the agent reads the initial HTML response and nothing more, which makes server-rendered or statically generated content the robust choice. You lose nothing by assuming it and you lose a lot by assuming otherwise and being wrong.
If your React app uses Next.js, Remix, or another framework with proper server-side rendering or static generation, and the main content is present when you view source on a deployed page, you are in good shape on this dimension. The check is empirical: load your page, view source, and search the raw HTML for a unique headline or body sentence. If it is there, any client that reads HTML can see it. If you see a `<div id="root">` and a script tag, you have work to do.
AI crawlers do not measure Core Web Vitals. The vitals are Google's real-user experience metrics, they feed Google Search page experience signals, and they describe what your human visitors get. We still check them for two reasons. A site with poor vitals almost always has server or markup problems that do affect how reliably an automated client can read it, and the person who clicks through from an AI answer has to use the page you actually built. We treat the web.dev thresholds (LCP under 2.5s, CLS under 0.1, INP under 200ms) as a floor.
Technical Health carries a 15% weight in our methodology, which is meaningful without being the largest dimension. In our model Citability (25%) and Brand Authority (20%) carry more, because they bear more directly on whether an engine has reason to recommend you. Technical Health gates them: a page that cannot be fetched and parsed gives the higher-weighted dimensions nothing to act on. These weights are ours, and we publish them so you can argue with them. See the SignalScore methodology page for the full breakdown and our background for why we built it this way.
Ready to improve your AI visibility?
Book a strategy call. We will audit your search and AI presence and recommend a plan tailored to your business.