Do AI crawlers actually read llms.txt and other AI-discovery files?

Based on our first-party server data, mostly no. Across 142,458 AI-agent visits to our site, the published discovery files (llms.txt, ai.txt, and the .well-known manifests) got 0.75% of the traffic combined. AI agents spent ~68% of their visits on ordinary content pages. This matches industry findings that ~97% of llms.txt files receive zero AI-bot requests.

What is the Open Knowledge Format (OKF)?

OKF is an open, vendor-neutral format published by Google Cloud in June 2026 for representing knowledge as a directory of markdown files with YAML frontmatter. It is an agent-context format, not a search-ranking signal. We use it to serve clean, cross-linked, machine-native versions of our press releases, entities, and topics that agents can extract and attribute.

Does adding OKF or an /okf/ endpoint improve AI citations?

We don't claim that yet — and you should be skeptical of anyone who does. OKF is not a documented ranking signal for ChatGPT, Perplexity, or Google. Our hypothesis is that exposing cleaner, attributable facts to the agents already reading your content improves extractability. We shipped it as a measured experiment and are tracking whether answer-time agents actually follow it over a three-week window.

How should I optimize my own site for AI crawlers in 2026?

Meet agents where they already are: your content. Make your key facts extractable (clear claims, statistics, structured data), keep answer-time crawlers like GPTBot and PerplexityBot unblocked in robots.txt and your CDN, and stop over-investing in self-published discovery files that the data shows agents ignore. Measure actual bot behavior in your server logs before trusting any 'AI optimization' checklist.

What 142K AI Bot Visits Taught Us About AI Discovery

Every "get cited by AI" checklist tells you to publish an llms.txt, an ai.txt, a .well-known/ai-plugin.json, a knowledge-graph file. We did all of that — we build AI-discovery infrastructure for a living. Then we did something the checklists don't: we looked at our own server logs to see whether AI agents actually fetch any of it.

They mostly don't. So we rebuilt our AI Discovery Protocol around what the data actually shows. This is that story — the numbers, the redesign, and an honest bet we're now measuring in the open.

The finding: AI agents ignore discovery files and read your content

We analyzed every AI-agent visit recorded on pressonify.ai. Here is what 142,458 all-time agent visits looked like:

The published AI-discovery files — llms.txt, ai.txt, ai-discovery.json, knowledge-graph.json, and the whole .well-known/ manifest cluster — received 0.75% of all agent traffic, combined.
llms.txt got 192 fetches. Ever. ai.txt got 2. The five .well-known agent manifests got 143 combined, across the site's entire history.
In the last 30 days, AI agents spent their visits like this: 50.3% on blog content, 11.8% on news pages, 5.5% on wiki pages — and 0.6% on discovery files.

The single sharpest number: our AI search platform comparison — one blog post — drew 3,167 agent hits in 30 days. That is roughly three times more agent traffic than our entire published-discovery-file surface has received in the site's whole history.

The agents doing the crawling are exactly who you'd expect: OpenAI (46% of agent traffic), followed by Amazon, Meta, Anthropic, and Perplexity. They are real, they are frequent — and they are reading our content, not our manifests.

This isn't unique to us. Independent log studies have found that ~97% of llms.txt files receive zero AI-bot requests. We just confirmed the same pattern on our own logs, at scale. It's the same lesson we've learned before, the hard way, more than once.

Why the "publish a discovery file" model quietly fails

The instinct behind AI-discovery files is reasonable: give the machines a clean, structured index so they don't have to parse your messy HTML. The problem is a chicken-and-egg one. An agent has to already know a file exists and choose to fetch it. Retrieval and answer-time systems don't spider your .well-known/ directory hoping for treasure — they land on the page a search returned, read it, and move on.

Google has been blunt about this for its own products. Its official AI-features documentation states there are "no additional requirements… nor other special optimizations necessary" beyond being indexed with a snippet — and specifically that there's "no special schema.org structured data that you need to add." Google publicly declined to support llms.txt, comparing it to the long-dead keywords meta tag.

So we had a surface of 20-plus self-published files scoring under 1% of agent attention, generating maintenance work, and — worst of all — creating the comfortable illusion that we were "AI-optimized." That illusion is the real cost. It's the same trap we've written about with the broader Citation Economy: measuring the thing that's easy to measure (files published, impressions served) instead of the thing that matters (whether an AI actually used your content).

The redesign: meet agents where they land

If agents live on your content pages, that's where the structured, citable knowledge belongs. So we rebuilt our AI Discovery Protocol around a simple inversion:

Stop asking agents to discover a file. Expose the knowledge from the page they're already on.

Concretely, we did three things.

1. We built a walkable knowledge surface at `/okf/`

Every page an agent lands on now carries a link to its own machine-native concept document. A news page points to /okf/pr/{slug}.md. A wiki page points to /okf/topic/{slug}.md. A blog post points to /okf/article/{slug}.md. These are served as Open Knowledge Format documents — clean markdown with structured YAML frontmatter — generated live from the same data behind the human page.

Crucially, the concepts cross-link to each other. A press-release concept links to the organization behind it; the organization links to every release about it; a topic links to its source press releases. An agent that follows one link can walk the graph — company → its announcements → the topics they touch — instead of re-parsing a dozen HTML pages. You can browse the root of it yourself at /okf/index.md.

2. We retired the files nobody fetched

The five deadest discovery files — ai.txt (2 lifetime hits), ai-manifest.json, and three unused .well-known manifests — now redirect to the live knowledge bundle. Fewer artifacts, less drift, no more pretending an endpoint with two hits ever is "discovery."

3. We instrumented it so we can tell the truth later

Every fetch of an /okf/ document is logged and split by bot type. That distinction matters: a training crawler fetching a file because it's linked is indexing, not using. The signal we care about is answer-time agents — the ChatGPT-User and Perplexity-User fetches that happen when a real person asks a real question — following the bridge.

What OKF actually is (and what it isn't)

The Open Knowledge Format is an open, vendor-neutral spec Google Cloud published in June 2026. It represents a body of knowledge as a directory of markdown files with a thin layer of YAML frontmatter — the same "LLM-wiki" pattern, standardized. Every concept declares a type, a title, a description, a canonical resource URL, tags, and a timestamp, and links to related concepts with ordinary markdown links.

Here's the honest part, because it's the part that matters and the part most "AI optimization" pitches skip: OKF is an agent-context format, not a search-ranking signal. No AI search engine has documented it as a citation input. If someone sells you "add OKF and get cited more," they're doing exactly what the llms.txt crowd did.

We didn't adopt OKF because it's magic. We adopted it because our data says agents consume our content, and the cheapest way to hand those agents a clean, complete, attributable version of our facts — with an explicit source URL on every concept and a walkable graph of related facts — is a standard format we can serve from the pages they already read. The format was almost free to add: our press releases already publish a markdown mirror, and our entities and wiki topics were already structured. OKF just gave it a common shape.

So why do we think this helps citations?

Not because OKF is a ranking lever. Because of a plainer mechanism.

When an answer-time agent retrieves your page to answer a question, the quality of the citation it produces depends on the quality of the facts it can extract. Extract from raw HTML and it gets your claim tangled in nav bars, share buttons, and cookie banners. Give it a concept document — "PlantGift eliminated shipping fees across Ireland and 25 EU countries on April 20, 2026," with the canonical /news/ URL attached and links to the company entity and related topics — and the model has your exact claim, cleanly, attributed to your domain, with context one hop away.

That's the hypothesis: maximizing extractability and provenance for the agents already reading you is a better path to being cited accurately than publishing files they don't fetch. It's the same principle behind everything in the Citation Economy — clear, structured, verifiable facts win citations; walls of unstructured prose don't.

The honest part: this is a measured experiment

We shipped this on July 7, 2026. We do not yet know if it works, and we're not going to tell you it does.

Here's the contract we set ourselves, in the open:

We watch answer-time /okf/ traffic for three weeks.
If answer-time agents don't follow the bridge — if OKF fetches stay near zero like ai.txt did — we kill it. It cost us little (we serialized data we already had), and we'll have learned something real.
We only invest further if the data validates it: meaningful, growing traffic from the answer-time crawlers of the major AI platforms.

That's the difference between AI-discovery infrastructure and AI-discovery theater. Theater ships a file and declares victory. Infrastructure ships an instrument and reads it. If this experiment fails, you'll read that here too — the same way we published our zero-baseline citation data and the search queries only AI agents type when the numbers weren't flattering.

What this means for your site

You don't need an /okf/ endpoint tomorrow. You need to stop optimizing for a model of AI crawling that your own logs would disprove. Three takeaways:

Read your server logs before trusting any AI checklist. Filter for AI user-agents (GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, meta-externalagent). See which paths they actually hit. We'd bet it's your content, not your manifests.
Invest where agents are. Make the facts on your high-traffic pages extractable: clear claims, real numbers, structured data that matches your visible text, and canonical URLs. Keep answer-time crawlers unblocked in robots.txt and your CDN — our robots.txt guide for AI crawlers covers exactly which agents to allow.
Measure citations, not artifacts. "Files published" is a vanity metric. "AI cited our brand as the answer" is the only one that pays. Run your content through our Citability Checker to see whether an AI can even extract your key facts before you worry about discovery files.

The AI-discovery layer of 2025 was a pile of published files. The evidence says the AI-discovery layer of 2026 is your content — structured well enough that when an agent reads it, your facts come out clean and attributed. We just rebuilt ours around that. Now we get to find out if we're right.

What 142K AI Bot Visits Taught Us About AI Discovery