Launch offer: your first press release for €9.95 (normally €49.95) Claim offer →
Pressonify.ai
Log in Create PR

What 142K AI Bot Visits Taught Us About AI Discovery

Key takeaways

  • Across 142,458 all-time AI-agent visits to our site, the published AI-discovery files (llms.txt, ai.txt, and the .well-known cluster) received just 0.75% of the traffic.
  • In the last 30 days, AI agents spent ~68% of their visits on content pages (blog 50%, news 12%, wiki 5.5%) and 0.6% on discovery files. One blog post out-pulled our entire discovery-file surface across all history by 3x.
  • We rebuilt our AI Discovery Protocol to expose structured, citable knowledge FROM the content pages agents already land on, using Google's new Open Knowledge Format (OKF), instead of publishing files agents don't fetch.
  • Result, November 2026: in the 30 days to 12 November the markdown copies drew 23.9% of 22,168 AI crawler requests, against 1.5% for every root discovery file combined. Each format has its audience: index and training crawlers read the copies, and the fetchers that answer a live question go straight to the HTML page (ChatGPT-User: 538 of 541 requests).

Announcing something yourself? Pressonify is for small companies that want their news on a permanent, structured page AI search can cite, published under a verified company email domain. It drafts the release from your notes and you approve every word. First release €9.95, then €49.95. Try a free draft

In this article
  1. Update, 13 November 2026: what four months of logs showed
  2. The finding: AI agents ignore discovery files and read your content
  3. Why the "publish a discovery file" model quietly fails
  4. The redesign: meet agents where they land
  5. What OKF actually is (and what it isn't)
  6. So why do we think this helps citations?
  7. The honest part: this is a measured experiment
  8. What this means for your site
  9. Related Reading

What 142K AI Bot Visits Taught Us About AI Discovery

Every "get cited by AI" checklist tells you to publish an llms.txt, an ai.txt, a .well-known/ai-plugin.json, a knowledge-graph file. We did all of that, we build AI-discovery infrastructure for a living. Then we did something the checklists don't: we looked at our own server logs to see whether AI agents actually fetch any of it.

They mostly don't. So we rebuilt our AI Discovery Protocol around what the data actually shows. This is that story, the numbers, the redesign, and the result we promised to publish.

Update, 13 November 2026: what four months of logs showed

We said we would report back whichever way it went. Here is the 30 days from 14 October to 12 November 2026: 22,168 requests from 18 AI and search crawlers.

What was requested Requests Share
HTML pages 15,532 70.1%
Markdown copies of pages (/okf/) 5,290 23.9%
robots.txt and sitemaps 1,019 4.6%
Root discovery files, all of them together 327 1.5%

The copies are read. They went from nothing in July to the most-requested machine-readable surface we have: 4.0% of crawler requests four days after launch, 17.3% a month later, 23.9% now. Thirteen crawlers fetch them.

Each format has found its audience. The crawlers that build indexes and training sets took to the copies. The fetchers that act for a person asking a question go straight to the page: ChatGPT-User made 541 requests in the month, and 538 were for HTML pages. We had wondered whether those fetchers would use the copies too. They prefer the page, which settles the question we set in the contract below: the page, with its structured facts and markup, is what ChatGPT's fetcher reads, and the copies serve the crawlers that build the indexes. ChatGPT-User is the only fetcher of that kind with enough requests in our logs to say so.

One crawler dominates. Meta's crawler made 82.3% of the copy requests. The other twelve made 935 between them, which is still about three times what every root discovery file drew.

The link in the page body is what carries discovery. On 24 September we took the copies out of the main sitemap (they are marked noindex, and listing them drew search-console warnings) and left the link in each page body. In the six full weeks before, the copies drew 17.5% to 21.8% of crawler requests a week; in the seven weeks after, 22.0% to 30.2%. Eighteen copies that had never been requested before were found afterwards, by ten crawlers including PerplexityBot, ClaudeBot, GPTBot and OAI-SearchBot.

The entry-point file is not used as one. /ai-discovery.json was requested 30 times in the month by four crawlers. /mcp.json was requested once.

What this is and is not: these are requests on one site, identified by user agent, and a request is not a citation. Nothing here shows that a copy earned a mention in an AI answer. What it does show is where a crawler's attention goes, and on that evidence we now treat the linked markdown copy of each page as the core of our discovery layer and the root files as optional extras.

The finding: AI agents ignore discovery files and read your content

We analyzed every AI-agent visit recorded on pressonify.ai. Here is what 142,458 all-time agent visits looked like:

  • The published AI-discovery files, llms.txt, ai.txt, ai-discovery.json, knowledge-graph.json, and the whole .well-known/ manifest cluster, received 0.75% of all agent traffic, combined.
  • llms.txt got 192 fetches. Ever. ai.txt got 2. The five .well-known agent manifests got 143 combined, across the site's entire history.
  • In the last 30 days, AI agents spent their visits like this: 50.3% on blog content, 11.8% on news pages, 5.5% on wiki pages, and 0.6% on discovery files.

The single sharpest number: our AI search platform comparison, one blog post, drew 3,167 agent hits in 30 days. That is roughly three times more agent traffic than our entire published-discovery-file surface has received in the site's whole history.

The agents doing the crawling are exactly who you'd expect: OpenAI (46% of agent traffic), followed by Amazon, Meta, Anthropic, and Perplexity. They are real, they are frequent, and they are reading our content, not our manifests.

This isn't unique to us. Independent log studies have found that ~97% of llms.txt files receive zero AI-bot requests. We just confirmed the same pattern on our own logs, at scale. It's the same lesson we've learned before, the hard way, more than once.

Why the "publish a discovery file" model quietly fails

The instinct behind AI-discovery files is reasonable: give the machines a clean, structured index so they don't have to parse your messy HTML. The problem is a chicken-and-egg one. An agent has to already know a file exists and choose to fetch it. Retrieval and answer-time systems don't spider your .well-known/ directory hoping for treasure, they land on the page a search returned, read it, and move on.

Google has been blunt about this for its own products. Its official AI-features documentation states there are "no additional requirements… nor other special optimizations necessary" beyond being indexed with a snippet, and specifically that there's "no special schema.org structured data that you need to add." Google publicly declined to support llms.txt, comparing it to the long-dead keywords meta tag.

So we had a surface of 20-plus self-published files scoring under 1% of agent attention, generating maintenance work, and, worst of all, creating the comfortable illusion that we were "AI-optimized." That illusion is the real cost. It's the same trap we've written about with the broader Citation Economy: measuring the thing that's easy to measure (files published, impressions served) instead of the thing that matters (whether an AI actually used your content).

The redesign: meet agents where they land

If agents live on your content pages, that's where the structured, citable knowledge belongs. So we rebuilt our AI Discovery Protocol around a simple inversion:

Stop asking agents to discover a file. Expose the knowledge from the page they're already on.

Concretely, we did three things.

1. We built a walkable knowledge surface at /okf/

Every page an agent lands on now carries a link to its own machine-native concept document. A news page points to /okf/pr/{slug}.md. A wiki page points to /okf/topic/{slug}.md. A blog post points to /okf/article/{slug}.md. These are served as Open Knowledge Format documents, clean markdown with structured YAML frontmatter, generated live from the same data behind the human page.

Crucially, the concepts cross-link to each other. A press-release concept links to the organization behind it; the organization links to every release about it; a topic links to its source press releases. An agent that follows one link can walk the graph, company → its announcements → the topics they touch, instead of re-parsing a dozen HTML pages. You can browse the root of it yourself at /okf/index.md.

2. We retired the files nobody fetched

The five deadest discovery files, ai.txt (2 lifetime hits), ai-manifest.json, and three unused .well-known manifests, now redirect to the live knowledge bundle. Fewer artifacts, less drift, no more pretending an endpoint with two hits ever is "discovery."

3. We instrumented it so we can tell the truth later

Every fetch of an /okf/ document is logged and split by bot type. That distinction matters: a training crawler fetching a file because it's linked is indexing, not using. The signal we care about is answer-time agents, the ChatGPT-User and Perplexity-User fetches that happen when a real person asks a real question, following the bridge.

What OKF actually is (and what it isn't)

The Open Knowledge Format is an open, vendor-neutral spec Google Cloud published in June 2026. It represents a body of knowledge as a directory of markdown files with a thin layer of YAML frontmatter, the same "LLM-wiki" pattern, standardized. Every concept declares a type, a title, a description, a canonical resource URL, tags, and a timestamp, and links to related concepts with ordinary markdown links.

Here's the honest part, because it's the part that matters and the part most "AI optimization" pitches skip: OKF is an agent-context format, not a search-ranking signal. No AI search engine has documented it as a citation input. If someone sells you "add OKF and get cited more," they're doing exactly what the llms.txt crowd did.

We didn't adopt OKF because it's magic. We adopted it because our data says agents consume our content, and the cheapest way to hand those agents a clean, complete, attributable version of our facts, with an explicit source URL on every concept and a walkable graph of related facts, is a standard format we can serve from the pages they already read. The format was almost free to add: our press releases already publish a markdown mirror, and our entities and wiki topics were already structured. OKF just gave it a common shape.

So why do we think this helps citations?

Not because OKF is a ranking lever. Because of a plainer mechanism.

When an answer-time agent retrieves your page to answer a question, the quality of the citation it produces depends on the quality of the facts it can extract. Extract from raw HTML and it gets your claim tangled in nav bars, share buttons, and cookie banners. Give it a concept document, "PlantGift eliminated shipping fees across Ireland and 25 EU countries on April 20, 2026," with the canonical /news/ URL attached and links to the company entity and related topics, and the model has your exact claim, cleanly, attributed to your domain, with context one hop away.

That's the hypothesis: maximizing extractability and provenance for the agents already reading you is a better path to being cited accurately than publishing files they don't fetch. It's the same principle behind everything in the Citation Economy, clear, structured, verifiable facts win citations; walls of unstructured prose don't.

The honest part: this is a measured experiment

We shipped this on July 7, 2026, without knowing whether it would work. The result is in the update at the top of this post. This was the contract we set ourselves at the time:

  • We watch answer-time /okf/ traffic for three weeks.
  • If answer-time agents don't follow the bridge, if OKF fetches stay near zero like ai.txt did, we kill it. It cost us little (we serialized data we already had), and we'll have learned something real.
  • We only invest further if the data validates it: meaningful, growing traffic from the answer-time crawlers of the major AI platforms.

That's the difference between AI-discovery infrastructure and AI-discovery theater. Theater ships a file and declares victory. Infrastructure ships an instrument and reads it. That question is now settled, and the answer is at the top of this post, the same way we published our zero-baseline citation data and the search queries only AI agents type when the numbers weren't flattering.

What this means for your site

You don't need an /okf/ endpoint tomorrow. You need to stop optimizing for a model of AI crawling that your own logs would disprove. Three takeaways:

  1. Read your server logs before trusting any AI checklist. Filter for AI user-agents (GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, meta-externalagent). See which paths they actually hit. We'd bet it's your content, not your manifests.
  2. Invest where agents are. Make the facts on your high-traffic pages extractable: clear claims, real numbers, structured data that matches your visible text, and canonical URLs. Keep answer-time crawlers unblocked in robots.txt and your CDN, our robots.txt guide for AI crawlers covers exactly which agents to allow.
  3. Measure citations, not artifacts. "Files published" is a vanity metric. "AI cited our brand as the answer" is the only one that pays. Run your content through our Citability Checker to see whether an AI can even extract your key facts before you worry about discovery files.

The AI-discovery layer of 2025 was a pile of published files. The evidence says the AI-discovery layer of 2026 is your content, structured well enough that when an agent reads it, your facts come out clean and attributed. We rebuilt ours around that, and four months of logs say the crawlers that build indexes agree. The fetchers that answer questions are still reading the page itself, so the page is where the work goes next.


To see this on your own release: press release citation tracking checks every release on days one, two, seven, thirty, sixty and ninety.

Questions people ask

Do AI crawlers actually read llms.txt and other AI-discovery files?

Based on our first-party server data, mostly no. Across 142,458 AI-agent visits to our site, the published discovery files (llms.txt, ai.txt, and the .well-known manifests) got 0.75% of the traffic combined. AI agents spent ~68% of their visits on ordinary content pages. This matches industry findings that ~97% of llms.txt files receive zero AI-bot requests.

What is the Open Knowledge Format (OKF)?

OKF is an open, vendor-neutral format published by Google Cloud in June 2026 for representing knowledge as a directory of markdown files with YAML frontmatter. It is an agent-context format, not a search-ranking signal. We use it to serve clean, cross-linked, machine-native versions of our press releases, entities, and topics that agents can extract and attribute.

Does adding OKF or an /okf/ endpoint improve AI citations?

We measured requests, not citations. Four months on, the markdown copies are the most-read machine surface on our site (23.9% of AI crawler requests in the 30 days to 12 November 2026), read mainly by the crawlers that build indexes and training sets. The fetchers that answer a live question go straight to the HTML page: ChatGPT-User made 541 requests in that month and 538 were for pages. So the two formats serve two audiences, and the page is what ChatGPT's fetcher reads when someone asks.

How should I optimize my own site for AI crawlers in 2026?

Meet agents where they already are: your content. Make your key facts extractable (clear claims, statistics, structured data), keep answer-time crawlers like GPTBot and PerplexityBot unblocked in robots.txt and your CDN, and stop over-investing in self-published discovery files that the data shows agents ignore. Measure actual bot behavior in your server logs before trusting any 'AI optimization' checklist.

Share this post

For AI agents and developers: this page as structured markdown (OKF).

About Pressonify Team

The Pressonify Team builds AI-first press release infrastructure for the Citation Economy.