11. Archive and gate

A review that exists only as terminal output is gone the moment the window closes. To be evidence it has to outlive the run, and to change behavior it has to be able to stop a merge. This rung is both.

Write the run down ¶

make report

wrote reports/gateway/review.results.json

That is a self-contained check-result document. It carries what produced it and at what build, which design and a content hash of that design, which tiers were attached, which rules actually ran, and every finding and outcome.

Each of those exists for a reason. Two results documents are only comparable once you know which build made each. The content hash is the revision identity, so a stale document cannot be silently read against a design that has since changed. And the list of rules that ran is what separates a clean design from a run that checked nothing, drawing the same distinction rung 9 was about.

Read it back without the design ¶

$ agni review designs/gateway --results-out review.results.json > /dev/null

$ agni results review.results.json --format markdown
# Review: Sample Board design review

Design: `designs/gateway`

**3 pass, 9 fail, 0 n/a, 2 not-automated, 1 provisional (of 15)**

The useful property is that this works with the design gone. Copy the JSON to a machine that has never seen the board, has no parameter corpus, and no profiles, and it renders the same report.

That is what makes it archival. A year from now, when the design file has moved and the tool has moved on several versions, the document still says what was checked and what was found.

A report a person can read ¶

The results document is for machines and for your future self. For a reviewer who wants to read the run today, render it as HTML:

$ agni check --format html designs/gateway/gateway.edn > review.html

That writes a self-contained page: failures first, then each rule with what it examined, why the rule exists, and what to do about a failure. It needs no server and no assets, so it attaches to a review ticket or a release folder as one file.

Give it --server self and each subject becomes a link into a viewer, which opens the board with that verdict’s proof drawn:

agni check --format html --server self designs/gateway/gateway.edn > review.html

That is one command rather than two, and the reason is worth understanding, because it is the same reason a link can be withheld. A link is a promise the reader can follow, so the CLI emits one only when it knows the server serves the design it just read. self starts a viewer on a free port over THIS run’s own mount table and blocks until Ctrl-C, so the two cannot disagree by construction.

Point it at a server someone else is running and the promise has to be checked instead:

agni check --format html --server http://localhost:8080 \
  --mount work=. designs/gateway/gateway.edn > review.html

The --mount is load-bearing there. The CLI asks that server whether it serves the same mounts from the same roots, and a mount you did not NAME is one it minted for this run alone, which means nothing on anybody’s server. Then the report renders plain text subjects instead of links, and says why. That is the honest answer rather than a URL that resolves on nobody’s machine.

For one design and a quick look, agni open serves it and prints a ready-made check command with the mount already filled in, which is the same trade in the other order: open when you want the viewer open anyway, --server self when you want the report.

Two things this is not. It is not the archival artifact: the results document replays without the design and this does not. And a rule that reports violations without stating what it examined is labelled “findings only” here, with its rows captioned so nobody reads silence from it as a clean bill. That caption is the difference between a report and a reassurance.

Gate a merge ¶

$ agni check designs/gateway/gateway.edn --conventions conventions.yaml --params params --fail-on error
exit 2

Exit 2 is a tripped gate, so CI fails. Put that one line in your pipeline and a board with an error-severity finding cannot merge.

The gate reads severity, not verdict ¶

Here is the part that surprises people. Rev B fixed the I2C pull-ups and the naming, and its review went from 8 failures to 6. Run the gate on it:

$ agni check designs/gateway/gateway-rev-b.edn --conventions conventions.yaml --params params --fail-on error
exit 2

Still failing. The remaining error is the datasheet finding on U2, the one the review reported as provisional because it rests on placeholder data.

--fail-on operates on finding severity, which is a statement about consequence. provisional is a statement about evidence quality. They are different axes, and a finding can be severe and poorly evidenced at the same time, and this one is both.

So a provisional finding still gates. Whether it should is your call, and there are two honest answers. Leave it gating and treat the block as pressure to go transcribe the real datasheet value, which is usually the right instinct. Or drop the parameter tier out of the gate command until the corpus is trustworthy, and accept that those checks are not gating yet:

$ agni check designs/gateway/gateway-rev-b.edn --conventions conventions.yaml --fail-on error
exit 2

Still 2, and the reason is worth stopping on: dropping the parameter tier removed the datasheet error, and a different one was underneath it: a CAN host declaring the interface without its STB signal. Narrowing what a gate can see does not make a board pass, it only changes which failure you are looking at. If you want to know what remains, run without --fail-on and read the list.

What you should not do is lower the severity of the rule to make the gate pass. That changes what the tool claims about consequence in order to change an exit code, and every future reader of that finding inherits the lie.

Gate on the checklist too ¶

The gate above cannot see one whole class of regression, and it is worth meeting before you rely on it. Run the review with a floor under how many items it has to answer:

$ agni review designs/gateway --min-answered 13
exit 0

Thirteen of the fifteen items get answered, so the floor holds. Now move the parameter corpus out of the way, as somebody reorganising a repository eventually will, and run exactly the same two commands you have been gating with:

$ mv params params-old
$ agni review designs/gateway --coverage
**13 of 15 covered**, **12 answered** — 2 pass, 10 fail, 1 n/a; 2 not-automated

Covered did not move. It is still 13 of 15. The item that used to check a part against its datasheet now reads not-applicable, because its rule is still in the catalog and merely has nothing to read, and not-applicable counts as covered. Nothing in the failure count says so either.

--min-answered is the number that moved, and it trips:

$ mv params params-old
$ agni review designs/gateway --min-answered 13
error: designs/gateway answered 12 of 15 checklist items, below --min-answered 13 (13 covered; an item whose rule is present but whose inputs are absent reads not-applicable, which counts as covered and not as answered)
exit 2

review therefore needs a gate of its own. check --fail-on asks how bad the answers were; this asks whether the questions were answered. A checklist quietly answering fewer of its own items looks identical to a clean board on every other number you have.

Put params back before continuing:

mv params-old params

Two notes on using it. A provisional does not trip --fail-on-outcome fail, because it is a failure resting on placeholder data and a pipeline that goes red on data quality is a pipeline somebody switches off. Ask for it by name when you want it: --fail-on-outcome fail,provisional. And a tripped gate exits 2 where a broken run exits 1, so a script can tell a bad board from a bad tool.

Where to start gating ¶

--fail-on error first. Errors are things that will not work at all, which almost nobody argues with, and the initial list is usually short.

Tighten to --fail-on warning once that list stays empty on its own. Going straight there on an existing board produces a wall of failures on day one, and the reliable outcome is that somebody turns the gate off.

Run the full review in CI alongside the gate and publish the results document as a build artifact. The gate answers whether this can merge. The document answers what was checked, and that is what you will want in six months when somebody asks whether a particular question was ever considered.

That is the ladder ¶

You now have a board that is read correctly, checked against general rules and your team’s own, with a checklist whose every item reports honestly, comparable revision to revision, archived, and gating.

The two things worth revisiting periodically are coverage and the parameter corpus. Coverage tells you how much of your checklist is really being decided. The corpus is usually the cheapest way to move it.