Measurement model
How INGEST decides what a machine can trust
INGEST is a comprehension harness, not an SEO score product. It asks whether machines can retrieve, parse, extract, verify, relate, and trust the important facts on a page — with evidence for every finding, and without inventing certainty.
1. The question
Search engines and AI systems do not see your brand the way a human does. They see bytes, redirects, HTML, scripts, structured data, and whatever survives policy and rendering. INGEST measures that substrate.
The product question is not “Is this optimized for AI?” It is: can a machine attribute important claims cleanly?
2. Measurement layers
Fetch & access
Default request plus public crawler identities (Googlebot, Bingbot, GPTBot, ClaudeBot, Perplexity, Applebot, Common Crawl, and related classes). Status, redirects, body bytes, and challenge signals are recorded per agent.
Robots policy
Matched group, matched rule, and allow/disallow result stay separate from HTTP usability. A 200 page can still be disallowed. An unreadable robots.txt remains UNKNOWN — never silently treated as allow or block.
Raw HTML vs rendered DOM
Title, canonical, H1, links, JSON-LD, and visible text are compared. JavaScript-added content is measured as dependency, not automatically graded as a defect.
Structured data
Every JSON-LD block is tested for standard parse, one-pass unescape repair, graph shape, and warnings. Parse validity is never treated as schema truth.
Atomic facts
Extracted propositions preserve subject, predicate, object, source text, assertion mode, polarity, quantifiers, temporality, and modality where detected. Interrogatives are not asserted facts. Under- extraction is preferred to unsupported inference.
Parity & entities
Visible vs machine claims are classified MATCHED, HUMAN_ONLY, MACHINE_ONLY, CONTRADICTORY, STALE, or AMBIGUOUS. Qualifier/scope differences do not auto-escalate to contradiction. Entity graphs stay partial when evidence is partial.
3. Hard separations
- Robots policy ≠ HTTP delivery ≠ content usability. Collapsing these hides the real failure mode.
- JSON parse OK ≠ schema correct. Missing names, breadcrumb mismatches, and weak identity remain findings.
- Question ≠ claim. FAQ questions are paired with answers; interrogative form does not become an asserted proposition.
- Hash-only noise ≠ differential delivery. Material fields (status, bytes, title, canonical, H1, usability, structured-data count) drive differential observation.
- UNKNOWN ≠ zero. Unmeasured components stay null. Coverage and composite score do not invent completeness.
4. Fail-closed extraction
When extraction is uncertain, INGEST prefers:
NO_FACT · UNKNOWN · AMBIGUOUS · INFERRED
over an explicit fact that required silent reconstruction. Trust hierarchy:
- Correct extraction
- Honest under-extraction
- Unsupported inference (rejected)
5. Coverage vs score
Measurement coverage reports how many leaf components were actually measured. A composite machine-ingestion score may exist alongside incomplete coverage. A high score with unmeasured freshness or search interpretation does not imply those were observed.
Term-level definitions: Coverage & scores glossary.
6. Evidence contract
Findings and atomic facts carry inspectable provenance where available:
- source URL
- representation / source type
- selector or location
- raw source text
- extraction method and assertion mode
- confidence tier
If a claim cannot point back to evidence, it is not treated as trustworthy.
7. Validity & regressions
Permanent fixtures and unit/integration markers lock behavior. Examples include BCP-47 locale handling, robots group selection, 404 usability, robots/HTTP separation, hash-noise differential delivery (REGRESSION_005), and interrogative-not-fact extraction (REGRESSION_006).
Local and CI both run npm run fixtures:validity. No measurement bug is fixed without a deterministic test.
8. What INGEST does not claim
- It does not reproduce private crawler internals.
- It does not decide which conflicting assertion is “true.”
- It does not treat cloaking labels as legal conclusions.
- It does not turn marketing language into unsupported service facts.
- It does not replace Search Console, SERP, or live AI-answer probing (deferred).
Run it
The consumer overview explains the stake. The instrument runs the measurement. Methodology is the contract between them.