Compliance Testing & Regression Suites

Unit tests tell you the code does what its author intended. They say nothing about whether a rule pack still produces the verdicts a planner agreed with last quarter, which is the only question anyone asks after a change. This module of Compliance Reporting & Audit Trail Generation covers the test corpus that answers it: how to build one, what to assert, and how to read the diff when a change moves a hundred verdicts and only twelve of them were meant to move.

Prerequisites #

• A pipeline that is already deterministic in the sense described in rule engine design: same inputs, same rule version, same verdicts. A non-deterministic pipeline cannot have a regression suite, only a source of noise.
• Snapshotted, hashed input layers, so a corpus run reads exactly what it read last time.
• Verdict records carrying their components — measurement, threshold, margin — since asserting only the outcome misses half the regressions.
• A place to store expected results that is version-controlled and diffable.
• A handful of parcels a planner has assessed by hand. These are worth more than any number of synthetic cases.

Three Layers of Test, Doing Different Jobs #

The word “test” covers three quite different activities here, and conflating them produces a suite that is slow, brittle and still misses things.

Measurement tests are ordinary unit tests over the measurement functions: given this parcel geometry and this reference line, the setback distance is 20.14 feet. They are fast, they need no rule pack, and they are where geometric edge cases belong — the corner lot, the flag lot, the curved frontage.

Rule tests assert that a single rule applied to a fixture parcel yields a stated verdict, including the boundary case at exactly the threshold. They exercise the operator, the tolerance and the three-way outcome, and they are the cheapest place to catch an inverted comparison.

Regression runs evaluate a corpus of real parcels under the current rule pack and diff the verdicts against a stored baseline. They are slower, they need snapshots, and they are the only layer that catches the interaction effects — a precedence change that quietly alters which rule governs, a geometry repair that shifts an area, a new overlay row that captures parcels nobody expected.

All three are worth having. Only the third answers “did this change do what we meant it to?”

Choosing the Corpus #

A regression corpus is a sample, and the sampling decides what the suite can see.

Include, deliberately: parcels whose verdicts a planner has reviewed and agreed with, because those are the only ground truth available; parcels sitting within a tolerance of a threshold, because they are where changes surface first; one parcel per awkward geometry — corner, flag, through, curved, split-zoned; parcels under each overlay and each combination of overlays that actually occurs; and parcels with recorded overrides of each kind.

Add a random sample on top of the deliberate one. Curated corpora drift towards the cases the team already thought about, and the random tail is what catches the case nobody imagined. A few hundred random parcels alongside a hundred chosen ones is a good working balance: small enough to run in minutes, broad enough to be surprised by.

What to exclude is anything whose expected verdict is unknown. A corpus entry whose baseline was produced by the pipeline itself and never checked by a human is a record of what the code did, not of what it should do — useful for detecting change, useless for detecting error. Keeping the two categories distinct in the corpus, and reporting them separately, keeps everyone honest about which kind of assurance a green run provides.

Reading a Regression Diff #

The output of a regression run is a list of parcels whose verdicts changed, and interpreting it is the skill this whole practice rests on.

Every change falls into one of four buckets. Intended — the amendment lowered a setback and these are the parcels it was meant to affect. Explained — a boundary refresh moved a district line, and the affected parcels are along it. Suspicious — a change with no plausible cause in the diff, which is where the bug is. And cosmetic — a margin that moved in the sixth decimal without changing an outcome, which should be filtered out before anyone reads the list.

The practice that makes this tractable is to require an expected blast radius before the change is made. “This amendment should change roughly forty parcels, all in R-2” is a prediction; a run that changes four hundred, half of them in C-1, has falsified it and the change does not merge. Teams that adopt this find it catches more real defects than the assertions do, because it tests the author’s understanding rather than the code’s behaviour.

def classify_changes(baseline, current, expected_rule_ids):
    """Split a verdict diff into the four buckets a reviewer needs."""
    changed = []
    for key, before in baseline.items():
        after = current.get(key)
        if after is None or after["outcome"] == before["outcome"]:
            continue
        bucket = ("intended" if before["rule_id"] in expected_rule_ids
                  else "suspicious")
        changed.append({**after, "was": before["outcome"], "bucket": bucket,
                        "margin_before": before["margin"], "margin_after": after["margin"]})
    return changed

What to Assert Beyond the Outcome #

A regression suite that compares only pass and fail misses roughly half of what changes, because a great many defects move a measurement without moving a verdict — until the day they do.

Assert the margin as well as the outcome, with a tolerance appropriate to the measurement. A change that moves every setback by two hundredths of a foot has not changed a verdict today and has changed something, and finding out why now is much cheaper than finding out when a parcel near a limit flips next quarter.

Assert the governing rule identity. A parcel that passes under a different rule than it did last month is a precedence or routing change, and the outcome alone conceals it entirely. This is the single most valuable assertion after the outcome itself, because routing changes are the least visible and the most surprising.

Assert the counts: parcels evaluated, verdicts produced, indeterminate outcomes. A run that produces the same verdicts for a hundred fewer parcels has lost a hundred parcels, and a suite comparing verdict-by-verdict will report no differences at all.

And assert the inputs: the snapshot hashes recorded in the baseline should match the ones the run used, or the comparison is between two different datasets and any difference is uninterpretable. Failing loudly on a snapshot mismatch prevents an entire category of wasted investigation.

def compare(baseline, current, margin_tol=1e-6):
    """Compare verdicts on outcome, margin and governing rule — not outcome alone."""
    diffs = []
    for key, before in baseline["verdicts"].items():
        after = current["verdicts"].get(key)
        if after is None:
            diffs.append({"key": key, "kind": "missing"})
            continue
        if after["outcome"] != before["outcome"]:
            diffs.append({"key": key, "kind": "outcome",
                          "from": before["outcome"], "to": after["outcome"]})
        elif after["rule_id"] != before["rule_id"]:
            diffs.append({"key": key, "kind": "governing_rule",
                          "from": before["rule_id"], "to": after["rule_id"]})
        elif abs(after["margin"] - before["margin"]) > margin_tol:
            diffs.append({"key": key, "kind": "margin",
                          "delta": after["margin"] - before["margin"]})
    return diffs

Classifying by kind rather than lumping everything into “changed” is what makes the output readable. A run reporting forty margin changes and no outcome changes is a very different situation from one reporting forty outcome changes, and a flat list obscures the distinction precisely when it matters.

Property-Based Tests for the Geometric Layer #

Fixtures test the cases somebody thought of. The geometric layer benefits from a complementary technique that tests cases nobody did.

Property-based testing generates geometry — random polygons, random parcel-and-line configurations — and asserts invariants that must hold regardless of the input. Several are genuinely useful here. A buildable envelope must always be contained by its parcel. A distance measurement must be symmetric and non-negative. Apportioned district shares must sum to one within tolerance. An area computed after repair must be within the delta budget of the area before, or the parcel must be flagged.

These invariants catch a specific class of defect that fixtures rarely reach: the pathological geometry — a parcel with a hundred near-collinear vertices, a polygon with a hole touching its shell — that no one would think to draw but that a county fabric contains dozens of. When a property test fails, it also hands you the minimal failing input, which is usually a far better bug report than a real parcel that happens to trigger the same thing.

They complement rather than replace the corpus. Invariants say the measurement is self-consistent; only planner-reviewed parcels say it is right. A suite with both is testing two different things, which is why running both is worth the modest extra time.

Keeping the Baseline Honest #

A stored baseline is only useful if updating it is a deliberate act.

Baselines should live in version control beside the rule pack, in a diffable text format, and updating one should be part of the same change that caused it — reviewed together, so a reviewer sees both the rule edit and the verdicts it moved. A baseline updated in a separate commit “to make the tests pass” is the mechanism by which a regression suite stops detecting regressions, and it is worth naming that explicitly in a team’s conventions.

The baseline also needs to record what produced it: the rule pack version, the input snapshot hashes, and the software versions. Without those, a diff between two runs conflates rule changes, data changes and library changes, and the resulting investigation goes nowhere. This is the same manifest discipline as provenance and lineage tracking, applied to tests.

Testing the Things That Are Not Rules #

A compliance pipeline has several behaviours that no rule describes and that a rule-focused suite will not exercise, each of which has caused a real incident somewhere.

The indeterminate path. A fixture whose measurement deliberately fails — a parcel with no derivable front lot line, a structure with no footprint — should produce an indeterminate verdict with a reason, not a pass and not an exception. This is the single most valuable non-rule test, because the failure it guards against is silent and consequential.

The empty-input path. Running with a constraint layer that is present but empty should fail loudly rather than reporting universal compliance. One fixture that supplies an empty layer and asserts a startup failure covers it permanently.

The fallback path. Where the pipeline is configured to fall back to an older snapshot when a source is unavailable, that path needs exercising deliberately, including the assertion that the report names the snapshot it used. Fallbacks are by definition rare in production and therefore untested by ordinary use.

Reconciliation. A test that runs a small corpus and asserts that parcels in, verdicts out and indeterminate counts reconcile catches the class of defect where a stage silently drops rows — which no per-parcel comparison can see, because the dropped parcels simply are not there to compare.

Determinism itself. Running the same corpus twice, in different orders and at two worker counts, and asserting identical output is a single test that guards every property this section depends on. It is slower than the others and worth running on every rule change regardless.

These five take an afternoon to write and they cover the failure modes that produce the worst outcomes — the confident wrong answer and the quietly incomplete run — which no amount of rule-level testing reaches.

None of them belongs in the regression corpus, incidentally: they are assertions about pipeline behaviour rather than about verdicts, and mixing them into a corpus run makes both harder to read. Keep them beside the rule tests, where they run on every commit and cost nothing.

Where the Suite Runs #

The three layers belong at different points, because their costs differ by orders of magnitude.

Measurement and rule tests run on every commit — they take seconds and need nothing but code. The regression run belongs on every change to the rule pack, the precedence table, the override store or the geometry pipeline, which is a smaller set of changes than “every commit” and a larger set than “every release”. And a scheduled full run against fresh snapshots catches the changes that arrive from outside: a republished parcel layer, a library upgrade in the base image, a PROJ grid that appeared or disappeared.

That last one is worth having even when nothing internal has changed, because it is the only thing that detects drift caused by the world rather than by the team. A weekly run reporting “no verdict changes” is a small, reassuring signal; the week it reports two hundred is the week you want to know before somebody else does.

Troubleshooting #

• The corpus run is slow enough that people skip it. It is too big. A few hundred parcels should run in minutes; if it does not, the pipeline’s per-parcel cost is the problem the suite has usefully surfaced.
• Every run shows dozens of cosmetic changes. Floating-point noise from order-dependent aggregation. Fix the determinism rather than widening the comparison tolerance, which would hide real changes too.
• The diff is empty after a rule change. Either the corpus contains no parcels the rule touches — a coverage gap worth fixing — or the rule was not actually loaded.
• A change is intended but the affected set is unfamiliar. Trust the diff over the intention. This is the case the practice exists for.
• Baselines conflict on every merge. Two changes both moved verdicts. Regenerate after merging rather than resolving line by line, and re-review the combined diff.

Finally, keep the suite’s own runtime honest. A regression run that takes an hour will be skipped under deadline pressure, and a skipped suite is worth nothing at all. If the corpus has grown past a few minutes, sample it rather than shrinking the assertions: a smaller set of parcels tested thoroughly catches more than a larger set tested loosely, and the full corpus can still run on a schedule where nobody is waiting for it.

Related #

Part of: Compliance reporting and audit trail generation

• Writing golden-file tests for zoning rule packs — the baseline mechanics.
• Detecting rule drift between pipeline releases — attributing a change to its cause.
• Automating zoning code version control with Git — the review path a regression run gates.
• Rule engine design for zoning and setback automation — the determinism this practice depends on.
• Provenance and lineage tracking — recording what produced a baseline.

Conclusion #

A compliance pipeline earns trust by being predictable, and predictability is a claim that has to be tested rather than asserted. Keep three layers: fast measurement tests for geometry, rule tests for operators and boundaries, and a regression corpus of real parcels — some planner-reviewed, some random — diffed against a version-controlled baseline. Predict the blast radius before a change and treat a surprise as a defect. Record the rule version, snapshots and library versions with every baseline. The result is a team that can amend a zoning code on a Tuesday and say exactly which parcels it moved.