Who respects your robots.txt? AI crawler compliance, measured in the open.
This site runs honeypot paths explicitly blocked in /robots.txt.
Every crawler that visits them is logged as a violation.
No guessing — just evidence.
Total Violations
—
Bots Observed
—
Verified (Web Bot Auth)
—
Bot Compliance Scoreboard
| Bot | Operator | Verdict | Signed | Violations | Visits |
|---|---|---|---|---|---|
| Loading… | |||||
robots.txt — honeypot config
# Only honeypot paths are blocked. The rest is open,
# so compliance with Disallow rules can be measured.
User-agent: *
Disallow: /do-not-crawl/
Disallow: /private/
Disallow: /honeypot/
Disallow: /training-data-forbidden/
Disallow: /no-ai-allowed/
Disallow: /robots-test/
visible link traps — linked & disallowed
Three honeypot paths are linked from this page and
also Disallow'd in
The other three honeypot paths (
/robots.txt. A
crawler that follows a link and visits the target has unambiguously
ignored the rule — a "via link" violation.
The other three honeypot paths (
/private/, /honeypot/,
/robots-test/) have no links anywhere on the site.
A hit there means the bot used the robots.txt
Disallow list as a crawl seed ("treasure map") or guessed the
path — a "via guess" violation.
// these links are for crawlers, not for you — clicking just adds noise
Live Violation Feed
No violations logged yet.
Discovery Reads — who fetches llms.txt, agents.md & grounding pages
These files are invitations, not traps:
/llms.txt is the LLM-oriented site summary, and
/agents.md (served at /AGENTS.md,
/agents.md and /.well-known/agents.md) is a
probe surface for agents that look for usage instructions. Reading them is
not a violation — it is the opposite signal: an agent
deliberately doing discovery. So far —
discovery reads have been logged.
| Bot | Operator | llms.txt | agents.md | grounding | Total | Last read |
|---|---|---|---|---|---|---|
| Loading… | ||||||