Access & robots

Robots policy (A / D / ?)

Whether robots.txt allows, disallows, or leaves UNKNOWN the agent’s request for this URL.

A = ALLOWED by a matched robots rule. D = DISALLOWED. ? = UNKNOWN (no matched directive, unreadable robots.txt, or inconclusive) — never silently treated as allow or deny.

Robots policy does not equal HTTP status and does not equal usability.

On the report Crawler access · robots column

Usable

Whether the HTTP response is classified as usable content — independent of robots.txt.

Usable comes from HTTP classification only. A page can be robots-disallowed and still usable as bytes, or allowed and unusable (challenge, soft-404 shape, etc.).

Green ● means usable. ✕ means not usable for ingestion purposes.

On the report Crawler access · usable

• Usable is not “Google will rank this.”

Blocked (composite)

Convenience composite: robots Disallow OR content not usable.

Used in some scoring inputs and tooltips. Prefer reading robots and usable columns separately when diagnosing.

On the report Tooltips / scoring inputs

Challenge · CAPTCHA · WAF hint

Heuristic friction signals observed on the response — not courtroom proof of intent.

Challenge: bot wall / access challenge markers. CAPTCHA: captcha-like body markers. WAF hint: CDN/WAF fingerprint hints.

These are observational. They explain why usability may fail; they are not legal cloaking conclusions.

On the report Crawler access · challenge / captcha / waf

Differential delivery

Material field divergence across crawler identities for the same URL.

Triggered by differences in status, final URL, bytes, title, canonical, H1, challenge, structured-data count, or usability — not by body-hash noise alone.

Observation ≠ cloaking verdict. Hash-only noise is excluded so caching noise does not invent drama.

On the report Crawler access · callout when true

• Not a legal finding of cloaking.

Crawler identities

Public crawler User-Agent classes used for the access matrix (search, AI, archive, other).

INGEST uses publicly documented agent strings. It does not claim to reproduce private crawler internals.

Default HTTP Client is included alongside named agents.

On the report Run details · Crawler identities · matrix rows