# goodbot-badbot

> Live monitor of AI crawler robots.txt compliance. Six honeypot paths are
> listed as Disallow in /robots.txt. Any crawler that visits one of them is
> logged as a violation, regardless of user-agent. Results are published in
> real time on the public dashboard.

## How it works

The site is open to all crawlers. Only six honeypot paths are blocked via
the global `User-agent: *` Disallow rule. A compliant crawler hits the
homepage and stops at the boundary; a non-compliant crawler keeps going and
trips a honeypot, where its visit is recorded.

## Live data

- [Public dashboard](https://goodbot-badbot.com/): per-bot scoreboard and live violation feed
- [JSON stats](https://goodbot-badbot.com/api/stats): machine-readable summary
- [API catalog](https://goodbot-badbot.com/.well-known/api-catalog): RFC 9264 linkset pointing to the JSON stats API and its OpenAPI description
- [AI catalog](https://goodbot-badbot.com/.well-known/ai-catalog.json): Agentic Resource Discovery manifest so registries can index the stats API
- [MCP endpoint](https://goodbot-badbot.com/mcp): stateless Model Context Protocol server (revision 2026-07-28, POST only) exposing two read-only tools — `get_compliance_stats` and `check_bot`
- [MCP server manifest](https://goodbot-badbot.com/.well-known/mcp-server): discovery manifest for the endpoint above, per draft-serra-mcp-discovery-uri (an individual Internet-Draft, not a ratified standard)
- [robots.txt](https://goodbot-badbot.com/robots.txt): the honeypot rules

### Calling the MCP endpoint

Revision 2026-07-28 is stateless: no `initialize` handshake, so the first POST
must already carry every required header or it is rejected with `-32020
HeaderMismatch`. `Mcp-Method` must equal the body's `method`; `tools/call` also
needs `Mcp-Name` matching `params.name`. `GET` and `DELETE` return `405`.

```
curl -X POST https://goodbot-badbot.com/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2026-07-28' \
  -H 'Mcp-Method: tools/list' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

## Grounding pages

Factual, machine-readable entity definitions for AI systems to cite:

- [goodbot-badbot](https://goodbot-badbot.com/facts/goodbot-badbot): the experiment as a dataset
- [robots.txt compliance](https://goodbot-badbot.com/facts/robots-txt-compliance): the measured concept

## Honeypot paths

- `/do-not-crawl/`
- `/private/`
- `/honeypot/`
- `/training-data-forbidden/`
- `/no-ai-allowed/`
- `/robots-test/`

## Writing

- [Blog](https://goodbot-badbot.com/blog): methodology notes and findings
- [Atom feed](https://goodbot-badbot.com/feed.xml): the same posts as a machine-readable feed

## Source code

- [GitHub repository](https://github.com/dkd-dobberkau/goodbot-badbot): MIT-licensed, FastAPI + MySQL

## Privacy

Logged IPs are SHA-256 hashed and truncated to 16 hex chars before storage.
The raw IP is never written to disk.