Conformance corpus

This directory contains the fixtures and expected outputs that any conforming implementation of UW Markdown self-certifies against. It is the canonical test corpus — analogous to the CommonMark spec corpus or OpenAPI's compliance suite.

The corpus is organized by conformance tier (see UW_PROTOCOL_v1.md Part II for tier definitions):

conformance/
├── tier-1-reader/      Parse + display, read-only
│   ├── fixtures/       Well-formed .uw.md files an implementer parses
│   ├── malformed/      Files exercising validator / integrity / policy codes
│   │                     (CC-NN, FV-NN, DQ-NN, INT-NN, POL-NN, META_*)
│   │                     plus optional <id>.policies.json siblings for
│   │                     POL-* fixtures
│   └── expected/       Expected JSON parses + display strings + chat-format renders
├── tier-2-editor/      Round-trip writes, supersede semantics
│   └── fixtures/       <scenario>/{before.uw.md, operation.json, after.uw.md}
│                         Optional siblings: context.json (EditContext),
│                         options.json (ApplyEditOptions, e.g. {integrity: true,
│                         maintainGaps: true}), expected-error.json (negative
│                         path — assert applyEdit rejects with a specific code)
├── tier-3-calc-host/   Custom calculation evaluation
│   ├── fixtures/       <scenario>/{deal.uw.md, calc.json, expected-result.json}
│   └── refinement/     <scenario>/{deal.uw.md, expected-graph.json}
│                         Exercises extractDependencyGraph() against a fixture
└── tier-4-agent-host/  AI agent layers producing write_uw_section calls
    ├── fixtures/       <scenario>/{before.uw.md, expected-after-shape.json}
    └── profile/        <scenario>/{expected-layer-profiles.json}
                          Asserts BANCROFT_LAYERS layer→consumed_profile contract

How to self-certify

1. Implement (or update) your tool against the published UW_PROTOCOL_v1.md.
2. For each tier you claim, run every fixture in that tier through your tool.
3. Compare your output to the corresponding expected/ output.

Tiers 1–3 use byte-exact comparison (after JSON canonicalization and volatile-field stripping where applicable). Tier 4 uses shape assertions — LLM nondeterminism makes byte equality impractical, so the expected output is a JSON Schema fragment describing the shape of an acceptable agent response.

Running the reference test runner

The reference implementation ships a runner at scripts/run-conformance.mjs that exercises the corpus against @uwmd/core. CI runs tiers 1–3 on every PR.

# Default: tiers 1, 2, 3
node scripts/run-conformance.mjs

# Specific tiers
node scripts/run-conformance.mjs --tier=1,3

# Tier 4 is operator-driven (lint-only — does not invoke an LLM)
node scripts/run-conformance.mjs --tier=4

# Bootstrap / refresh expected outputs from the current library
node scripts/run-conformance.mjs --update

# Machine-readable output
node scripts/run-conformance.mjs --json

Volatile fields stripped before byte comparison: last_modified, _meta.timestamp, ts= fence attributes, and _meta.content_hash values (hashes canonicalize over the timestamp, so they vary per run even when content is stable). These change every run and are not normative.

Adding a fixture

Fixtures are normative once merged. To add one:

1. Open a PR with the fixture file and its expected output.
2. Explain in the PR description what scenario the fixture covers and why it isn't already covered by existing fixtures.
3. Reviewers will run the fixture against @uwmd/core (the reference implementation) and against at least one third-party implementation to confirm the expected output is correct, not just what the reference happens to produce.

See ../CONTRIBUTING.md for the full contribution process.

Regenerating expected outputs

The expected/ files are generated by running the reference CLI against the fixtures. To regenerate after a non-breaking improvement:

node scripts/regen-conformance.mjs

If a regeneration changes any expected output, that's a normative change to the protocol — call it out explicitly in the PR description and bump the protocol version in UW_PROTOCOL_v1.md.