Everyone is telling you to add an llms.txt file for “AI SEO.” The pitch is clean: drop a plain-text map of your best content at the root of your domain, and the large language models (LLMs) behind ChatGPT, Gemini, and Claude will read it and understand your site. It has the shape of a good idea. It costs almost nothing to add. So people are adding it on faith, by the tens of thousands.
I run a few websites, this one included, partly so I keep my hands on new tools as they land. So I added the file too. Then I did the thing I have been asking e-commerce teams to do for twenty years: I checked the logs before I believed the story. Usually I am the one evangelizing digital and performance analytics, but the tools are still catching up to this AI-agent world, so this time it was back to the raw logs.
My llms.txt file got zero reads from AI agents. Not one.
That is one site over one window, so I went looking for the pattern, and the pattern is worse for the file than my single log. Ahrefs analyzed 137,210 domains and found that 97 percent of valid llms.txt files received no requests at all in a month, and that roughly three-quarters of the bots that did fetch one were not AI tools at all. People are racing to install a thing that, by the evidence, almost nothing reads.
Meanwhile the crawlers that ignored my llms.txt read my actual pages just fine. In that same window Googlebot alone pulled well over a hundred full-content fetches, all HTTP 200, and a spread of others got clean 200s on my articles too: ChatGPT-User, Bingbot, Applebot, Meta’s crawler, and ByteDance’s Bytespider. My writing is legible to the ecosystem. It just is not legible because of the trendy file. It is legible because of the boring fundamentals, which I will get to.
Bots are the new front door
This is not vanity metrics. The front door to your content is no longer a human typing your URL. Increasingly it is an agent reading on someone’s behalf and then answering, recommending, or transacting. And this is not a forecast anymore. ChatGPT can now complete a purchase inside the chat, through Instant Checkout, from merchants who adopted the protocol, and the shopper never lands on the store. If the agent can read and act on your content, you are in the transaction. If it cannot, the agent buys from the competitor it could read, and the customer never sees you. That is disintermediation, live, not a slide.
I have been circling this for a while. I have written about how many e-commerce sites are effectively invisible to AI because their content only assembles in the browser, about why AI shopping agents behave like ghosts passing through storefronts, and about why AI agents will need a new kind of account management. This post is the plumbing underneath all of it: before an agent can transact on your behalf, it has to be able to read the page at all.
So “can AI agents actually read my site?” is not academic. It is the new version of “am I in the search index?” And like that older question, the honest answer comes from your logs, not from a vendor’s slide.
The llms.txt gold rush does not survive contact with the data
I want to be fair rather than absolutist, because llms.txt is not useless. It has a real niche, and John Mueller of Google’s Search Relations team named it himself when he called the file a temporary crutch to save some tokens for AI coding tools parsing developer documentation. That squares with what I see: coding assistants like Cursor and Copilot, and research frameworks like LangChain and LlamaIndex, do read it when a developer explicitly points a tool at your site. That is a genuine use. It is just not the “the models will discover and rank you” use it is being sold as.
And do not wave this away with “it is early, give it time.” Standards do take time, but this one has a structural problem that time does not fix. llms.txt is self-declared and unverifiable, which is exactly why Mueller compared it to the old keywords meta tag: a field where you tell the machine what you are, and the machine learns not to trust it. The standards that are getting real traction are the ones that are authenticated and verifiable, which is the opposite property. As of mid-2026, no major AI company has said it reads llms.txt in production.
So do not cargo-cult AI SEO. Adding a file because a blog post said to, without checking whether anything reads it, is the kind of hype-chasing I try to keep teams away from. Show me the evidence.
What actually got my content read
Here is the unglamorous truth about why the crawlers had no trouble with my site. None of it was novel. All of it was fundamentals.
Clean, server-rendered HyperText Markup Language (HTML), so the content is in the initial response and not locked behind client-side JavaScript that a crawler may never execute. Stable, readable URLs. Real document structure with proper headings. A sitemap. A permissive robots.txt that does not accidentally slam the door on the exact agents you want reading you. That is the whole recipe. It is boring, it is proven, and it is what put my essays in front of the crawlers without a single trendy file.
Here is the part that should worry a brand rather than reassure it: this is easy, and most storefronts still fail it. The reassurance (“just do the fundamentals”) and the alarm (“you are about to be disintermediated”) are the same sentence. The sites that get read are the ones that did the boring work. The ones that render everything in JavaScript, bury content behind interaction, or over-block their robots.txt are invisible to the agent doing the shopping, and they will not find out from a dashboard. They will find out from a quarter of missing demand they cannot explain. The fundamentals that made you legible to Googlebot in 2010 are the same fundamentals that make you legible to the shopping agent in 2026. The clients changed. The discipline did not.
A short debugging aside, because it is the trap most teams will fall into. Partway through this, Google’s Gemini told me, through its live-browsing feature, that it could not read one of my pages, and reported another as “blocked.” My first instinct was the wrong one: my server must be blocking Google. I nearly spent a weekend hardening it. Then I read the logs. My server had delivered every page it was asked for at HTTP 200, in full. The pages Gemini called “blocked” were not in my logs at all, no request, no error, because Gemini’s own tool had failed to send them and labeled its own failure as my fault. When an AI tool tells you it cannot read your content, that is a hypothesis, not a diagnosis. Confirm it against your logs before you touch a line of code.
Stay on top of what is being built right now
None of this is static. The standards for how agents discover, read, and transact are being written this year, and they are the authenticated, verifiable kind that llms.txt is not. OpenAI and Stripe shipped the Agentic Commerce Protocol that powers ChatGPT Instant Checkout. Google is building the buyer side too, with its Agent Payments Protocol and Universal Cart guidance for merchants. And the Model Context Protocol, the standard for how agents connect to tools and data, is now under neutral foundation governance with adoption across the major labs. This is the phase SEO was in decades ago, when the rules were being set and the teams paying attention got a durable edge. Track these the way your team once tracked search-algorithm updates, because the equivalent updates for agents are happening now.
I have watched this problem from a specific seat. As Chief Product Officer at Yottaa, a web-performance company for e-commerce, I spent years on the reality of bot and agent traffic hitting storefronts, including the work of telling the “known bots” apart so the good ones got through and the abusive ones did not. Tooling like Known Agents (formerly Dark Visitors) now lets any site owner see and govern crawler and agent traffic for free, so you do not need an enterprise platform just to look. If you do want the enterprise version of getting your storefront fast and legible to agents at scale, that is exactly what Yottaa’s Web Performance Services are for. But start by looking.
How to actually check whether agents can read your site
No faith, no vendor slides. If you have server access, do this yourself:
- Open your server access logs, or your content delivery network (CDN) logs if you sit behind one.
- Filter for the AI and search user-agents (UAs): Googlebot, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Bytespider, Applebot, Amazonbot, Bingbot, and CCBot, among others.
- Confirm they are getting HTTP 200 on your real content pages, not 403s, 404s, or redirect loops.
- Check your robots.txt is not blocking the agents you actually want reading you.
- If a specific AI tool claims it cannot read a page, do not assume your site is broken. Find that request in the logs and see what your server actually returned. The failure is often on the tool’s end.
If you are a VP and do not have shell access to Apache logs, the move is one message to your team: “Pull a week of access logs, filter for the AI crawler user-agents, and tell me if any of them are getting anything other than 200 on our product and content pages.” Or install a free crawler-visibility tool like Known Agents and read it yourself. Either way, the point is to look at what happened, not to guess.
“But bots can fake the user-agent string”
Yes, they can, and if you read that checklist and thought exactly that, good. A user-agent is self-reported text; anything can call itself Googlebot. So be clear about what this check is and is not. It is not a security control. If you are trying to keep malicious scrapers out, the user-agent is worthless and you need verified crawler IP ranges (the major crawlers publish theirs), rate limiting, and a firewall. Do not gate access on a UA string.
But that is a different job from this one. The question here is not “who is lying to me,” it is “are the legitimate agents I want to reach actually able to read my content.” For that, the honest crawlers are the signal, and the honest ones identify themselves truthfully because they want to be recognized. A spoofer pretending to be GPTBot does not change whether the real GPTBot got a clean 200 off your article. Verifying identity by IP is the next layer if you also care about the liars. It does not diminish the value of first confirming the real ones are getting through.
That is the whole discipline, and it is the same one that has served me for two decades. Measure, do not assume. I only know any of this because I read my own logs. You should read yours.