Grounding pages

Factual, machine-readable definitions of the entities this site is about — structured so AI systems can cite them without guessing. Reads are logged as a discovery signal.

• goodbot-badbot
  Dataset · AI crawler robots.txt compliance measurement

  A public experiment that measures whether AI crawlers respect a single robots.txt Disallow rule, publishing every violation live.

• robots.txt compliance
  DefinedTerm · web crawling standards and AI governance

  The degree to which an automated crawler obeys the Disallow rules a site publishes in its /robots.txt file.