Evidence

Measuring, testing, and trusting a result

The expensive mistakes here have not been wrong code. They have been correct-looking results, and nobody could have falsified them. A sweep found nothing because the detector was broken, nobody checked a rate for precision, and a green test could not have gone red. This page collects them.

Everything on it is one question in different clothes.

flowchart LR
    R(["a result you believe"]) --> Q{"what would this have looked<br/>like if it were wrong?"}
    Q -->|"different"| E["evidence"]
    Q -->|"the same"| N["not evidence yet"]
    N --> P["plant a known instance,<br/>then measure again"]
    P --> R
    classDef bad stroke-dasharray: 4 3;
    class N bad;

Measuring, and trusting a measurement

The fifteen rules below are grouped by what was doing the lying.

The instrument

1. A NEGATIVE RESULT NEEDS A POSITIVE CONTROL. “Zero hits across 62 documents” is a claim about the instrument until you show the instrument can find a known instance. Three absence claims here were artifacts of the detector: a table shape the heuristic could not see, a regex that could not match V CC, and a corpus sweep running a different code path from the one shipping. Plant a known instance first. The fixtures usually already contain one.

Two habits make that cheap. Instrument the gates, counting what each filter REJECTED and not only what it matched. And exercise the shipped configuration, because a sweep run with different flags has validated a different program.

The same detector run twice over the same corpus. Run as-is it reports zero hits, which is equally consistent with a clean corpus and with a blind instrument, so nothing distinguishes them. Run again with one known instance planted in the corpus, it reports one hit, and only then does a zero over the rest carry information. as run 62 documents the detector 0 hits none known to match the corpus is clean, or the instrument is blind, and nothing tells them apart with a positive control 62 documents the detector plus 1 planted instance 1 hit the instrument can see one, so a 0 over the other 62 is now a result a negative result is a claim about the instrument until the instrument has been shown to work

2. A CHECK RUN BEFORE THE EDIT IS NOT EVIDENCE ABOUT THE EDIT. The instrument can be right, the invocation can be right, and the run can still be stale. It happened before the code existed, and it is then cited as evidence about that code. A tsc --noEmit passed, two more files were written, the gate then failed on one of them, and the pass was reported as proof that the gate and the local check “resolve differently”. They do not, since all three spellings of that command catch the error, and re-running it would have said so in two seconds. Nothing about the output looks stale.

The cheap habit is to re-run the check as the LAST thing before you believe it. When a local pass and a gate failure disagree, suspect the ordering before you theorise about the tools. The theory is the expensive answer and it is almost never the right one.

3. A REPLICA OF THE ALGORITHM MEASURES THE REPLICA. Agni issue 682 reported that a capture’s freshness stamp treated a fetched board as no fixture at all, with a table of three stamps showing two of them identical. The table was produced by re-implementing the hash rather than calling it. The shipped hash has a guard that refuses a fixture with no tracked files outright, and the re-implementation left it out. The ticket’s conclusion survived in spirit, because the fetched corpus genuinely could not be used as a fixture. Every stated fact about the mechanism was wrong. Re-implementing is tempting whenever the real function is unexported or wants a whole harness. In exactly those cases, write the throwaway test that calls the real function. The one here took four lines and would have printed the refusal.

4. A detector that FIRES is a claim about the instrument too. A sweep for “does this document print a document number” reported 86% coverage on a regex that accepted TPS22918 and TCAN1145, both PART numbers, while missing SLVSAG5, a real one. Wrong in both directions, and the headline would have justified building an extractor on it. Run a new detector past known positives AND known negatives by hand before quoting a rate. When it cannot tell the two apart, that is the finding.

5. A SUBSTRING count is not an occurrence count, and it fails upward. Sizing the glossary sweep meant counting how often each term appears. A case-insensitive substring match reported TP 47 times in one section. The word-boundary count there is ZERO, because the match was finding tp inside http and output. The same sweep read pull-up at 89 and ESD at 43. Excluding rule names like i2c-pull-up and fenced code roughly halved both, to 44 and 13. Every error ran the same way, inflating the work. Scoping off the first numbers would have oversized the job about twofold. Worse, it would have recorded a term as well covered when the course never mentions it. Anchor the pattern, strip code and identifiers, and re-count before the number decides anything.

The fixture

6. A feature premised on a house CONVENTION needs its base rate measured on real designs, and the fixtures cannot tell you. Every fixture here names things the way the built-in vocabulary expects. That naming made the fixtures work, and it also makes a convention-shaped feature always look well-founded against them. One issue was written around a net-name convention that encodes endpoints. That convention covered 1.1% and 0.06% of nets on two real boards, and the tutorial project does not use it at all. Measuring the base rate takes one query, and it has twice changed what was worth building.

7. A feature no fixture EXERCISES cannot fail a test, and that reads exactly like working. The datasheet role tier shipped against a corpus where not one seeded spec declared pins. So every test passed, the real boards were unchanged, and nothing could have gone red if it were wrong. The fix was to give the shipped fixture the data the feature reads. The feature then went from unfalsifiable to demonstrable, with 0 rails becoming 3 on the tutorial netlist.

8. A fixture written to test a behaviour may be unable to EXHIBIT it, and then the finding is about the fixture. hier_bus_root.kicad_sch labels its bus DATA[1:0]. A session concluded from it that KiCad keeps bus members apart across a sheet boundary, and that conclusion held up a fix for a while. KiCad’s vector syntax is [first..last], so that fixture contains no bus at all and could never have shown members crossing. Change nothing but the spelling and kicad-cli joins them. The fixture is kept, with a comment saying it is really a control. A sibling fixture holds the real case, and the pair is only meaningful together. Before trusting a negative result from a fixture, check that a POSITIVE one was reachable from it.

9. Ask the reference tool rather than reasoning about the format. In one session three defects each turned on a detail that no reasonable derivation got right. Bus members pair off by ascending index and not in written order, a symbol is rotated before it is mirrored, and a colon-spelled range is not a bus. Sign conventions, a Y-flip and two composition orders were all in play. Build the smallest input that distinguishes the cases, run it through the tool that owns the format, and read the answer off the output. A schematic with labelled wire stubs at the compass points reports where a pin ended up. One with a tap per bus member reports which members crossed.

What reading cannot see

10. Reading the code is not running it. Three passes through one read path did not reveal that no producer fills SourceDoc.title as its contract specifies. Opening the app and looking at the field showed it immediately, and invalidated part of a PR merged an hour earlier. Review catches a layer that is wrong. It does not catch layers that are each internally consistent and collectively wrong, because every file reads fine on its own.

11. A claim about LAYOUT needs a paint-level check, not a rectangle comparison. getBoundingClientRect knows nothing about an ancestor clipping the box, so a child of a scrolling cell reports coordinates far outside that cell while being perfectly contained on screen. Reading those numbers as “it still overflows” is wrong in the safe direction. Ask the document instead, since elementFromPoint inside the NEIGHBOURING element returns whatever actually paints there. An A/B that flips the rules on a live page has to re-measure in each layout, since restoring the old rules reflows the row and moves the element you were probing.

Three table cells in a row. A wide badge strip sits in the middle cell, which scrolls, so the strip's box extends well past the cell's right edge while only the part inside the cell ever paints. getBoundingClientRect reports the full box and reads as an overflow. elementFromPoint, probed inside the neighbouring cell, returns the neighbour, because nothing of the strip paints there. one table row, and the middle cell scrolls a neighbour a neighbour overflow-x: auto the strip, painting clipped, never paints probe here getBoundingClientRect() returns the whole box, out to the dashed edge, so the strip reads as sitting far outside its cell elementFromPoint() at the probe returns the NEIGHBOUR, because nothing of the strip paints there, which is the question that was actually being asked the rectangle is correct and the conclusion is wrong: a clipping ancestor is invisible to it

When the world moved underneath

12. When a run contradicts your PREDICTION, the contradiction is the finding. A test was written to prove that a malformed project.yaml fails the run. It came back saying the run had SUCCEEDED. Chasing the gap, rather than blaming the fixture, showed that the descriptor was never reachable at all. Agni issue 312 was really about that unreachable descriptor, and not about the two call sites it named.

13. A long-lived ticket’s PREMISE erodes silently. Three substantial issues in one month had aged out before anyone picked them up. One had mostly shipped already. One assumed a convention that the only real boards contradicted. One had been done in pieces under other work. Nothing was wrong with any of them when filed. Adjacent work moved underneath while the ticket text kept asserting the old world. Read the comment thread as well as the body.

14. A ratio needs its DENOMINATOR constrained, not just its numerator. A query counted the passives with exactly one probe-covered net. The number was plausible, within a few percent of an independent tool’s, and the query read correctly in English. It was wrong. The query constrained how many nets were COVERED and never how many nets the part HAD. So it also matched parts with only one net, where that net was covered, and multi-pin arrays with one net of four covered. Two opposite errors came from one missing clause.

Nothing internal could have caught it, because the query returns rows, the rows are all real parts, and the count is stable across runs. Only a second tool’s answer falsified it. The general form reaches past coverage. Whenever a result is “the things where N of M hold”, a filter on N alone silently admits every row whose M is not what you assumed. State M as a clause even when you believe every subject has the same one.

15. A positive control catches a fixture that satisfies part of a check. A test asserted that a complete directory passes validation, and it failed. The check wants six files, and the fixture listed four of them. Had the fixture been used only in the NEGATIVE test, it would have failed for the missing files and read exactly like the guard working. A fixture that is supposed to pass has to actually pass.

Trusting a test

A new test is a measurement too, and its positive control is the red-check. A red-check neutralises a behaviour and confirms that the test covering it fails. The cheap way is to stash ONE tracked file (git stash push -- path/to/file.go, run, pop).

flowchart TB
    S["neutralise the behaviour<br/>git stash push -- one/file.go"] --> R{"run the test"}
    R -->|"red, naming the defect"| OK["the test is real"]
    R -->|"[build failed]"| B["proves nothing, and looks like it did.<br/>The stash took a declaration too.<br/>Revert the BEHAVIOUR in place instead:<br/>flip the branch to if false."]
    R -->|"green"| G["the test asserts nothing.<br/>Replace the heuristic with the PROPERTY."]
    classDef bad stroke-dasharray: 4 3;
    class B,G bad;

A compile error is not a failing assertion, so read the red output before believing it. The green branch has a worked example. A render test was built on a gap heuristic (“the two notes must be at least N apart”), and it passed with the fix removed. Another mechanism had shrunk the fixture enough that the gap held either way. The test now asserts the PROPERTY instead. It adds a marker at the same anchor and asserts that the two render at the same y, and it failed with the actual defect named. Run the red-check on every new test, not only the ones you doubt.

The neutralisation is code too, and it can silently do nothing. A red-check here reported that the test correctly found no difference, and that read as “the check is decorative”. The edit had not applied, because the string it replaced was not in the file. So the mutation was a no-op, and the run measured an untouched tree. If a red-check does not confirm that its own edit applied, it is a green run wearing a red hat. Assert the substitution (assert old in s, grep -c the new text) before believing what the suite then says.

Perturb one WIRING POINT at a time when a change spans several. Take a feature wired through a pass, a loader field, a config plumbing hop and an ordering. It invites one big neutralisation, and then the suite goes red for reasons nobody separates. Four separate perturbations cost four runs, and they tell you more than one run can. Neutering the datasheet class pass failed the pass, loader and CLI tests. Cutting the read option out of Overlay.ReadOptions failed ONLY the CLI test. Replacing the shared ordering with set order failed ONLY the glyph tests. That pattern maps each wiring point to the tests guarding it. If perturbing a wiring point fails nothing, that point has no test at all.

A comparison of two files that do not exist reads as a difference. cmp -s a b on two missing paths exits non-zero, and a script that prints “MOVED” on non-zero then reports a change nobody made. It happens when a command was supposed to write the files and wrote nothing. agni render --report -o f.svg does that, since it prints the report and ignores -o. So both sides of the comparison were absent, and the run still exited 0. Stat the inputs before trusting a comparison, or compare content you have actually read.

Count WHICH tests flip, not merely that the suite went red. Neutering reviewGate.trip() failed 6 of 8 gate tests. The two survivors were the opt-in guard and the flag-parse guard, and neither has any business depending on trip. “The suite went red” would have been equally true if the wrong six had failed. Naming which ones should survive costs one sentence of thought.

Ways a test passes while asserting nothing 

flowchart LR
    G(["a test that passes"]) --> W["...and could not<br/>have failed"]
    W --> A["the fixture never creates<br/>the condition"]
    W --> B["the probe is in the<br/>wrong place"]
    W --> C["the assertion was already true<br/>before the action"]
    W --> D["the oracle is the code<br/>under test"]
    W --> E["the sweep is empty"]
    W --> F["both sides run through<br/>the same helper"]
    W --> H["the meaning changed and<br/>the compiler said nothing"]
    W --> I["the baseline run did not<br/>build, and you grepped for<br/>matches"]

That last one is about MEASURING a change rather than testing it, and it produces a confident number out of nothing. You can compare a branch against main by stashing one file, but that gives a baseline only if the tree still compiles. A test file elsewhere may reference a symbol defined in the stashed file, and then the build breaks. A pipeline that greps the output for differences then reports none. Three such comparisons in a row came back clean and were all empty. Build the baseline in a git worktree at the commit you mean, where nothing can half-apply, and check that the run produced output at all before reading it.

The shape The case that taught it What makes it real
The fixture never creates the condition Two versions of one browser test went green with the containment CSS deleted. The public fixtures top out at three sheets, so with short names nothing overflowed whatever the rules said. Squeeze the column to its 48px minimum by dragging its own grip.
The probe is in the wrong place The second version still passed, because it probed the neighbouring cell’s CENTRE. The escaping chip reached 66px past its own cell while the neighbour was 301px wide, so the probe sat 84px clear of the bleed. Bleed arrives at the BOUNDARY and fades. Probe just inside the edge nearest the offender.
innerText does not mean visible An assertion written expect(await locator.innerText()).toBe("...") passed with display: none on the element, because the spec falls back to textContent for a node that is not rendered. The comment above it claimed it tested visibility. Assert a non-zero boundingBox(), or read the computed style.
The assertion was already true before the action A composition test clicks a finding and asserts expect(called).toContain("HighlightSheet"). It passes with the handler unwired, because deep-link restore highlights during boot, so it asserts that the page booted. Count across the action (before and after filtered lengths) or clear the log first. Every “did X happen” assertion over a running system has this shape.
The oracle is the code under test if skipRefDes(x) { t.Error(...) } went green under a deliberately broken skipRefDes, while its siblings written against literals went red. The test and the code agree by construction. Assert against literals, or against a set the production path produced.
The round trip goes through the forgiving half TestWriteRoundTripsIR read an EDIF file, wrote it, read it back and compared, and was still green through four defects that made the output unreadable to anyone else. Our reader resolves references after parsing the whole file, takes any token as an identifier, and guesses a port reference is a pin designator when no mapping says otherwise. The writer leaned on all three, and the test could not see it because the same reader closed the loop. Assert the emitted TEXT, or bring in an implementation that did not grow up with yours. Both, if the format is one other tools read.
The sweep is empty A catalog-wide test iterating “every rule that sets this field” passes trivially when no rule sets it. Count the rules it asserted over and fail at zero. A positive control belongs IN the test, not beside it.
The fixture encodes the rule’s own assumption Two i2c-pull-up fixtures gave their “pull-up” resistor exactly one net, no second end and no rail, so a rule correctly requiring a rail turned them red. Complete the fixture. Loosening the check until it passes again looks identical from inside the failing run.
The counter-example survives for the wrong reason A considered-set test asserted that a NO_CONNECT pin was OUT of a rule’s domain, and still could not tell a supply-scoped domain from one sweeping in every pin on the part, because the NC pin sat on no net and dropped out of both. A fixture case differing from the failing one in EXACTLY the property under test, here a signal pin alone on its own net.

These five need their own room rather than a row.

1. A test can compare a function against itself. A parity test held a rule’s Eval to its verdict projection, VerdictsToFindings(Eval(m)) against Eval(m). For a converted rule those are the same call, so it ran one function against itself and would have passed for any projection including a broken one. It now rebuilds the expectation independently, keeping the failing verdicts and taking their findings. The tell is structural rather than about the assertion. If both sides of an equality flow through the same helper, the helper is untested no matter how the assertion reads.

2. When two types carry the same field names, the compiler cannot help you. check.Verdict and check.Finding both have Subject, Kind, Pin and NetID, so when Rule.Eval moved from returning findings to returning verdicts, every

for _, f := range rule.Eval(m) { got[f.Subject] = true }

kept building and quietly started counting passes as failures. Two tests were already wrong this way and passed only by accident, because their rules were still wrapped in check.FailuresOnly. To find this, take each call site that the compiler did NOT complain about, and ask whether its meaning survived. Use Rule.Findings(ctx, m) where a test means violations.

3. scrollTop is always 0 under jsdom, so a scroll assertion passes with the bug present. jsdom has no layout engine, so nothing scrolls and nothing has a size. A panel that threw the reader back to the top on every click would have satisfied expect(el.scrollTop).toBe(prev) perfectly. Assert the CAUSE instead. Here the cause is that the rows were recreated. Node identity across the action (expect(after[i]).toBe(before[i])) is visible in jsdom, and it goes red on the real defect. A symptom needing a browser often has a cause that does not, and the cause is the better assertion anyway because it names why.

4. A test that SKIPS asserts nothing, and reads like a pass. A browser case was written with if (findings.length !== 1) t.Skip(...), to skip when the fixture did not produce the state it needed. It reported PASS in the suite output and had never executed its assertions. Skips are for a genuinely absent capability (no browser on the machine). They are never for a fixture that did not produce the state you wanted. That is a setup bug, and skipping hides it behind the same green tick as a real pass. Find the input that reaches the branch, or the test does not exist.

5. A -run filter that misses the test you care about still prints ok. go test -run "Supersed" ran three tests in cmd/agni and passed all three, so the package reported ok. It never ran TestCheckReportsAProjectsOwnSupersessions, because the filter is an unanchored regexp over the test name and Supersessions carries Superses rather than Supersed. The zero-match case is the safe one, since Go says ok ... [no tests to run] when a pattern matches nothing at all. A PARTIAL match lies, because the passes it reports are real and belong to other tests. Run a new test by its exact name once and read the --- PASS line carrying that name, before trusting any filtered run that claims to include it.

An out-of-tree oracle, for a format someone else reads

A round trip proves the two halves of ONE implementation agree. That is worth having, but it is not portability. Agni issue 580 sat in the gap between the two claims. It had four defects that the round trip could not see, and each was fatal to a reader not written alongside our writer.

Two oracles have been worth running, and both are free.

kicad-cli answers questions about what a KiCad file MEANS. Export the netlist, read the (net ...) blocks, compare which pins share a net. The technique is to ask rather than reason. Build a fixture that isolates exactly the question, and let the answer decide the code. Twice now the answer has been the opposite of the obvious one. A vector bus crossing a sheet boundary pairs its members by bit POSITION, so a child’s bit 0 joins the parent’s A0 whatever the parent’s range is spelled. A group bus pairs by member NAME, so two buses of equal width with different member names join nothing at all. Guessing either would have shorted unrelated signals while moving the net count the flattering way. readers/kicad/testdata/*.oracle are those answers, committed beside their fixtures with the regenerate command in each header, so the expected result is not one we chose.

GNU Electric answers whether a file we WROTE is one another tool can read. Its EDIF reader is an independent implementation with source bundled in the jar. Its batch mode asserts before reaching the importer, so drive it through BeanShell, which runs inside the initialized database:

curl -O https://ftp.gnu.org/gnu/electric/electric-9.08.1.jar
curl -o bsh.jar https://repo1.maven.org/maven2/org/apache-extras/beanshell/bsh/2.0b6/bsh-2.0b6.jar
java -Dagni.edif=<file>.edn -classpath electric-9.08.1.jar:bsh.jar \
     com.sun.electric.Electric -batch -s check.bsh

where check.bsh reaches the importer by reflection on EDIF$EDIFPreferences.doInput and prints the cell, node and arc counts. Passing -s removes the script from the argument list, and so avoids the batch-mode assertion. Electric prints a bogus 0 errors summary after printing every Error, line, so count the lines rather than trusting the total.

Neither belongs in the gate, since one needs a 19MB corpus and the other a 23MB Java download. The gate holds a test over the emitted text instead, readers/formats/e2e_edif_conformance_test.go. It asserts each PROPERTY the out-of-tree oracle established. The oracle says which properties are the right ones; the committed test keeps them true.

Read a foreign tool’s diagnostics as evidence about the FILE and not about the standard. Electric takes a cell rename’s display string as the cell’s name where ours takes the identifier, and reads exactly one token as a design’s name where the grammar allows a rename. The first is a real disagreement worth conceding, since a name it rejects is a file nobody can load. The second costs nothing. Loading the same file both ways gave identical counts, and conceding it would have meant discarding the design’s name.

Text you cannot match literally

Source What moves Match with
prototext an unstable extra space after a colon, inserted ON PURPOSE to discourage byte-comparison : +, or parse the proto
protojson one or two spaces after a colon, so two runs over one message differ parsed values, or strings.Fields joined by a single space
symbol text out of a doc-IR flattened subscripts, so VCCA arrives as V CCA (~850 occurrences in one corpus) a pattern tolerating the injected space
html/template output quotes, angle brackets and ampersands become entities, so "gateway" renders as &#34;gateway&#34; a fragment carrying none of them, or unescape before comparing

One before/after table was built with a one-space prototext pattern. It read ZERO for every “before” and was nearly shipped. The number itself was not the tell. The tell was an internal contradiction, a file showing 0 typed pins and 2 supply pins at once. Distrust any count whose parts do not add up. The injected space has bitten three times in unrelated places: a prose sweep, the derive pin path where it would have produced pin ids no symbol library could match, and in-document search.

Under EmitUnpopulated, a newly added field appears as its zero value in every existing consumer’s output. Adding one to a response message is additive on the wire and visible in the JSON, so “no data changed” and “byte-identical output” are different claims and only the first survives.

When the escaping row fails, the failure points the wrong way. Say a test asserts that a rendered page carries a sentence. It fails when the sentence contains a quote, and it fails by reporting the sentence absent. That reads as the feature not working, rather than as the assertion being unmatchable. So the debugging pass goes to the renderer, which is correct, instead of to the test. Assert on a fragment with no quotes in it, or unescape first. go test printing FAIL for a build failure belongs to the same family. In both, the mechanism reports the wrong thing as broken.