Technical SEO for a Web
Read by Machines Too

Because the failure is total rather than gradual. In classic search, a slow or partially-rendered page might rank a bit lower. In AI search, a blocked crawler or empty render means you're absent from the surface entirely — there's no degraded position to fall back to.

It's also the most common problem we find. A large share of brands with poor AI visibility have a purely technical cause, not a content one.

Check your robots.txt for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended directives, and check whether your WAF or bot-management tool blocks unknown agents by default. Then fetch a page as one of those agents and compare the response to what a browser shows.

That last step catches the rendering failures, which are far more common than outright blocks and much easier to miss.

It's a genuine business decision, not a technical default. Blocking them protects content from being used without attribution; allowing them is a precondition for being cited at all.

Most brands selling something benefit from being found. Publishers monetising content directly have a real case for restricting. We'll walk through the tradeoff rather than assuming — and if you decide to block, that's a legitimate choice, but you should make it deliberately.

Only if it renders client-side. React itself is fine; serving an empty HTML shell that populates via JavaScript is the problem, because several AI crawlers don't execute JavaScript.

Server-side rendering, static generation, or pre-rendering all solve it. Next.js, Nuxt, and similar frameworks support this natively — it's usually a configuration decision rather than a rewrite.

Access fixes can register within days to a couple of weeks, on the next crawl cycle. Rendering changes typically take two to six weeks to be fully reflected as pages are re-crawled.

It's the fastest-moving work we do, which is why we sequence it first in nearly every engagement.

Both, depending on what you need. We write developer-ready specifications your team can implement, and we can implement directly where you'd rather we did and access allows.

For WordPress and most common stacks we can usually handle implementation. For complex enterprise environments, we typically specify and support your engineering team rather than working in production ourselves.

We implement it because it's trivial and harmless, but we won't oversell it. Adoption is genuinely uncertain and it currently does far less than robots.txt directives and rendering strategy.

Treat it as a cheap hedge, not a strategy. Any agency positioning llms.txt as a core service line is dressing up a fifteen-minute task.

Where it's the primary need, we scope it as a fixed-fee project, typically in the low five figures for diagnosis plus specification. Implementation cost depends on whether we're doing it or your team is.

Within a broader retainer it's included as the first phase, since almost nothing else works until it's resolved.