What Official U.S. State Statutory Sources Tell AI Crawlers — August 2026

Version 2026.1 · Published 2026-08-16 · Coverage 2026-08-16/2026-08-16

A first-party, reproducible measurement of what each U.S. jurisdiction's official statutory website publishes in its /robots.txt about fourteen named AI crawlers retrieving a citable statute section. Of 47 official sources, 44 publish no rule naming any AI crawler. Three do — Alaska, the District of Columbia and Idaho — and two of those three carry an identical eight-token list alongside a content delivery network's own crawler token, which is a platform default rather than a drafted policy. Fourteen of the 47 publish no robots.txt at all, four could not be read on this date and are recorded as unverified rather than permitted, and one, California, instructs every automated client to stay out entirely, Googlebot included; 42 of 47 permit Googlebot to reach the cited section. This measures published policy, not behaviour and not accessibility: robots.txt is a request an obedient client honours, and the absence of a rule is the absence of a decision rather than a decision to permit.

Measured quantity: what the robots.txt published by a U.S. jurisdiction's official statutory website instructs a named AI crawler to do about the cited statute section URL

Downloads

Plain-text citation · BibTeX · Machine-readable release metadata

Methodology

  1. The population is the 47 U.S. jurisdictions for which an official statutory section URL was available to the auditor, carried over unchanged from the 2026-08-15 machine-readability release so that the two are directly comparable. The four jurisdictions whose only recorded URL is a third-party publisher are excluded from every total.
  2. For each jurisdiction, exactly one HTTP GET was issued, to /robots.txt on the scheme-and-host origin of that jurisdiction's section URL. No statute page was requested by this instrument at all: the preceding release already measured retrievability, and robots.txt alone answers the policy question.
  3. Requests used an identifying User-Agent carrying a contact URL, followed redirects, timed out at 25 seconds, and were separated by at least five seconds. No block, rate limit or exclusion was circumvented.
  4. Each robots.txt was parsed into user-agent groups per RFC 9309: consecutive User-agent lines share one rule block, and blank or comment lines do not terminate a group.
  5. For each of the sixteen user-agent tokens tested, the applicable group was selected by longest case-insensitive prefix match on the token, falling back to the wildcard group only where no token matched. This is what makes Applebot-Extended select its own group where one exists rather than inherit Applebot's.
  6. Where a crawler has no group of its own, prefix inheritance applies: a rule written for Applebot governs Applebot-Extended in the absence of a rule naming Applebot-Extended. This is the documented behaviour of the protocol as implemented by major crawlers, and it is an interpretive choice that changes verdicts, so the data records separately, for every jurisdiction, which AI tokens the file NAMES and which ones merely receive a verdict. A verdict inherited from a broader rule is not evidence that the publisher considered that crawler.
  7. Within the applicable group, the longest matching path pattern decides, and Allow wins an exact-length tie. Patterns anchor at the start of the path, '*' matches any run of characters, a trailing '$' anchors the end, and an empty Disallow value is an explicit allow-all that matches nothing.
  8. A jurisdiction whose host returned 404 or 410 for /robots.txt, or returned an HTML document instead of a robots file, is recorded as publishing no robots.txt, and every crawler is recorded as allowed by default with that basis stated.
  9. A jurisdiction whose /robots.txt could not be read at all — connection failure, block, or server error — is recorded as could_not_verify for every agent. That is never counted as allowed and never counted as a policy.
  10. Two controls had to pass or nothing would have been written. The parser control runs eleven synthetic robots.txt fixtures covering group selection, case-insensitive agent matching, longest-path precedence, Allow-beats-Disallow ties, '$' anchoring, '*' wildcards, empty Disallow, and consecutive User-agent lines. The live control requires that at least one audited host returned a parseable robots.txt, so that a total network failure reads as a broken check rather than as a finding about publishers.

Coverage and limitations

Data dictionary

VariableTypeDescription
jurisdictionstringTwo-letter postal abbreviation for the jurisdiction.
namestringJurisdiction name.
sectionstringThe citable statutory section whose URL supplies the host and path evaluated for this jurisdiction.
hoststringHostname of the official statutory source whose robots.txt was read.
robots_statestringrobots_txt_present, no_robots_txt (404/410), no_robots_txt_html_returned (an HTML document served at /robots.txt), unreadable_blocked (401/403/429), unreadable_connection_failed, unreadable_server_error, or unreadable_other_status.
robots_http_statusintegerHTTP status returned for the /robots.txt request; 0 where no response was received.
ai_tokens_namedstringSpace-separated list of the tested AI crawler tokens that appear literally in a User-agent line of this file. An empty value means the file names none of them.
ai_tokens_disallowedstringSpace-separated list of the tested AI crawler tokens whose applicable rule disallows the cited section path, whether by a rule naming that crawler or by the wildcard group that governs it.
GPTBotstringVerdict for OpenAI's training crawler on the cited section path: allowed, disallowed, or could_not_verify. The same column definition applies to every other user-agent column in this file.
*stringVerdict for the wildcard group — the rule that governs every crawler for which no specific rule was written. This is the baseline, not an AI crawler.
GooglebotstringVerdict for Google's search crawler, included as a reference point for whether the site permits automated retrieval of its law at all. Not an AI crawler.

Sources and provenance

License and citation

This release is available under Creative Commons Attribution 4.0 International. Attribution identifies the source; LawyerLand does not require a followed hyperlink as a condition of access.

LawyerLand (2026-08-16). What Official U.S. State Statutory Sources Tell AI Crawlers — August 2026 (Version 2026.1) [Data set]. https://www.lawyerland.com/research/ai-crawler-policy-state-statute-sources-2026-08/

Version history

This is version 2026.1. Prior version directories remain immutable. Material corrections receive a new version and release-feed entry.

Press and data contact · Release feed