Who respects your robots.txt? AI crawler compliance, measured in the open.

Discovery Reads — who fetches llms.txt, agents.md, grounding pages, the API/AI catalogs & the MCP manifest and endpoint

These files are invitations, not traps: /llms.txt is the LLM-oriented site summary, and /agents.md (served at /AGENTS.md, /agents.md and /.well-known/agents.md) is a probe surface for agents that look for usage instructions. The /.well-known/api-catalog goes one step further: it offers a machine-readable description of the JSON stats API, so an agent can use the interface instead of scraping this page. The /.well-known/ai-catalog.json Agentic Resource Discovery manifest goes further still — it lets agent-facing registries index this site so agents can find it in the first place. Finally /mcp is a stateless Model Context Protocol server: not a document to read but an interface to call, offering the same compliance data as two read-only tools. MCP has no ratified discovery standard, and for most of this experiment nothing told an agent that the path existed — how one found it was itself the measurement. Since August 2026 the site also serves /.well-known/mcp-server, the manifest proposed by draft-serra-mcp-discovery-uri — an individual Internet-Draft with no IETF standing that expires in September 2026. Whether anything ever reads it is the newest quiet column here. Reading or calling any of them is not a violation — it is the opposite signal: an agent deliberately doing discovery. So far — discovery reads have been logged, plus — HEAD probes.

Who is actually reading. Every caller is sorted into one class, because lumping them together was actively misleading: the single Unidentified row this table used to show accounted for 92 % of all discovery reads, and almost none of it was an AI agent. AI crawler is a bot from the registry this site tracks. Non-AI crawler is a self-declared SEO, search-engine or scanning bot — legitimate, just not what is being measured here. HTTP client is a scripted request (curl, httpx, Go) that names a library and nothing else; it is the most ambiguous class, since an LLM agent driving a script looks exactly like a shell one-liner. Browser UA means only that the string claims to be a browser — not that a person was driving it. Several of the heaviest grounding readers in that class rotate through 20+ IP addresses at roughly one request each and also walk straight into the honeypots. This site's own deploy and verification requests are excluded entirely; measuring them would be measuring ourselves.

Reads vs. probes. A HEAD request asks whether a file exists without taking the bytes — a crawler checking freshness or testing the waters, not consuming the content. Those are counted separately in the head probes column and are excluded from every per-surface read count, so the two are never conflated. Note that until August 2026 this site answered HEAD with 405 everywhere, so the probe column necessarily starts empty.

Invitation & trap — the callers that do both

A grounding page is the one thing this site hands out freely: a factual page, linked openly, where a read is a positive signal. A honeypot is the opposite — a path forbidden in /robots.txt and linked nowhere a human would click. They are meant to separate two kinds of visitor. Crossing them per caller shows how often they do not: — of grounding-page reads come from callers that also walked into a trap (— of — reads, from — of — callers).

The column that gives the game away is IPs. A caller with one address is a crawler doing its job; a "browser" with 25 reads spread across 25 addresses is a proxy pool. Note the clean rows too — they are the reason this table is not a blocklist. Named crawlers that read the facts and never touched a trap are doing exactly what the invitation asks, and several of the heaviest readers here are in that group. Anonymous callers have no name to print, so they are listed by user-agent; hover for the full string.