Testing

jaala’s hosts trust its answers enough to act on them, so we test most answers against a reference rather than against what a test author expected. Four layers of checks run on every change.

Every query, three ways ¶

Every test that evaluates a query runs it through Naive, SemiNaive{WrittenOrder: true} and the planned SemiNaive{}, and fails if they disagree. Written order has to match Naive’s rows and errors exactly. The planned evaluator has to match Naive’s rows, and give Naive’s error whenever it fails. Each query runs again under CanonicalCites(), where all three have to give the same rows, citations and witnesses (provenance). So a new optimisation is tested by every test that already exists, the moment it lands.

Generated programs ¶

Hand-written tests only reach the shapes their authors thought of. A generator writes random programs over random graphs, with recursion, negation, aggregates, constants, bound goals and Bind, and runs each through all three evaluators, plainly and under Witnesses(). The test shrinks a disagreement to a small program and graph before it reports it, which makes a report pretty quick to read. Its first run found four bugs the hand-written tests had missed (#87). The test lists known, filed disagreements, each with a program that reproduces it, so a fixed one shows up as fixed. Each program also runs over a Source that answers lookups (#126) and over a federation of three such stores that index only their first argument (#211, #154), and must answer as it does read whole. About a third of the graphs give their weights units, some all volts and some a mix (#192). The others’ programs compute now and then, in comparisons and as ?y = expr, sometimes written before what binds their operands (#209); its first long run found the planner running a multiplication backwards, which answers nothing for a factor of zero (#273). CI runs a few hundred programs on every change, and ./selfcheck.sh runs thousands.

Against Soufflé ¶

Every check above compares jaala with itself, so none of them can tell when the reference evaluator is wrong in the same way as the others. Soufflé is an independent Datalog engine, so the generated programs it can also express are translated into its language, run there, and compared with Naive’s answers (#23). The translator is test code, so Soufflé never becomes a dependency of jaala. It covers recursion, negation, comparisons, arithmetic, count, sum, min and max (by bindings or distinct), having, order by and limit, and bound goals. A program outside that is counted by its reason and not compared: list, which Soufflé has no form of, a division, since Soufflé’s numbers here are integers, and a rule head that reduces with no group column, where jaala derives an empty value Soufflé can’t hold in a number column. That leaves about three programs in four compared. Writing it found bugs in the translator, not in jaala, and one in Soufflé 2.5, which crashes on a rule whose body is two aggregates and nothing else. So the translator gives each aggregate a rule of its own. A test breaks the translation on purpose (dropping negation) and requires the comparison to catch it and shrink it, so a quiet run means the programs agreed rather than that nothing was checked. CI installs Soufflé in a job of its own and compares a thousand programs, and ./selfcheck.sh compares as many as it generates when souffle is on the path. When it isn’t, it says Soufflé couldn’t run, rather than passing.

Fuzzed query text ¶

The generator only writes programs that are well formed, so text a person or a pasted snippet might send never reaches the engine through it. Go’s fuzzer makes that text up instead (#89). One target requires that any input either parses or fails with a query: error, never a panic, and that a parsed rule prints back as text that parses to the same rule. The other evaluates whatever parses: no panic, every error keeps its query: prefix, and when the reference evaluator answers within a small budget, all three evaluators have to agree, as they do on the generated programs. Its first hour found a parser panic, a type declaration the printer wrote back as text the parser refused, a did-you-mean that took most of a second on a long name, a negative zero the index couldn’t find, and five programs the evaluators answered differently. All five are fixed, the last being a number spelled two ways (#148), which an answer now writes one way whichever spelling an evaluator met first. Each input it finds is fixed and kept as a seed, so go test replays it on every change, and ./selfcheck.sh fuzzes each target for a while longer.

Work baselines ¶

Answers being right says nothing about cost. Seventeen workloads (closures, same-generation, a points-to analysis, a netlist the size of an agni board, and four of those again under CanonicalCites()) record the work the planned evaluator does, and a test fails when any of them moves more than 10% either way (#88). Work is counted, not timed, so it’s the same on every run and on every machine. The baselines found two costs worth fixing as soon as they existed (#96, #97).

Store costs are measured separately, since a store’s round trips and scans aren’t the engine’s work. BenchmarkStores federates 1, 10 and 50 repo-shaped stores, each answering lookups from the patterns it indexes and scanning for the rest, and reports for a few host-shaped questions the stores asked, the calls made to them, the calls an index couldn’t answer, and what asking again calls (#218). It is the exercise of the store federation mission, and its smoke test runs in CI.

The docs ¶

This site is checked the same way. Every example runs on the engine when the site is built, pins its answer or its error, and fails the build if the engine disagrees. The live editor’s engine is tested as wasm under Node. The error catalogue is checked against every message in jaala’s source, and every internal link and anchor in the built site is followed. Writing these pages found bugs too: a panic on a goal with too few arguments (#133), an internal name in an error (#127), and blank aggregates (#122).

Tests that can fail ¶

A test that can’t fail checks nothing, so every new test is red-checked: break the behaviour it guards, and confirm it fails on its assertion, not on a build error. Where a fixture might not tell the cases apart, the test carries a control that proves it does, for example that a workload is big enough for a cost regression to show. To find what the tests miss, we inject bugs into the rewrites on purpose. One that reused variable names across inlined calls passed every hand-written test, and the generated programs caught it.