Writing Golden-File Tests for Zoning Rule Packs

A rule pack is data, and data has no unit tests — which is why a one-character change to a threshold can ship without any test failing. A golden-file test closes that gap: a fixed set of parcels, evaluated by the current pack, compared against an approved baseline. Every change to a rule then shows up as a diff someone approves or rejects. This guide builds that suite, choosing fixtures that exercise the edges rather than the typical case, storing baselines that produce a readable diff, and keeping approval meaningful. It is the baseline mechanics behind compliance testing and regression suites.

Prerequisites

Step-by-step

Step 1: Choose fixtures that exercise the boundaries

A fixture set of typical parcels tests almost nothing, because typical parcels are far from every threshold and stay compliant under any plausible bug. Choose deliberately for the cases that discriminate.

FIXTURE_CRITERIA = [
    ("just_compliant",    "measured value within 1% above the threshold"),
    ("just_violating",    "measured value within 1% below the threshold"),
    ("corner_lot",        "two frontages — the envelope case most often wrong"),
    ("split_zoned",       "two districts on one parcel"),
    ("overlay_partial",   "partly within a protective overlay"),
    ("irregular",         "concave, so centroid tests misbehave"),
    ("tiny",              "below the minimum lot size"),
    ("no_frontage",       "landlocked — many rules cannot apply"),
    ("null_attributes",   "missing dwelling units and missing lot width"),
]

Twenty to fifty parcels chosen this way discriminate better than ten thousand chosen at random, and they stay reviewable — which matters, because a diff nobody reads is not a test.

Fixtures chosen to discriminate, not to be representativeEach fixture criterion against the failure it is there to catch.CatchesJust compliant / justviolatingA one-unit threshold change; a unit errorCorner lotA second frontage silently not appliedSplit zonedApportionment replaced by majority-governsIrregular, concaveCentroid tests placing a parcel outside itselfNull attributesMissing treated as zero
Twenty parcels chosen this way discriminate better than ten thousand chosen at random — and they stay reviewable, which is the whole mechanism.

Step 2: Freeze the inputs completely

A golden test is only meaningful if the only thing that can change the output is the thing under test. Everything else — parcel geometry, district assignment, library versions, the clock — has to be pinned.

import geopandas as gpd

FIXTURE_PATH = "tests/fixtures/parcels.geoparquet"
RUN_DATE = "2026-01-15"          # fixed: rules can have effective dates

def load_fixtures():
    gdf = gpd.read_parquet(FIXTURE_PATH)
    assert gdf.crs.to_epsg() == 2263, "fixture CRS changed — baselines are invalid"
    assert len(gdf) == 34, "fixture set changed — regenerate baselines deliberately"
    return gdf

Asserting the fixture count and CRS inside the loader is a small thing that prevents a large confusion: a fixture set that quietly grew produces baseline diffs on parcels that were never reviewed, and the diff then looks like a rule change.

Step 3: Serialise the baseline so its diff is readable

The baseline format determines whether review actually happens. A single-line JSON blob technically records the result and produces a diff nobody can read; sorted, one record per line, with rounded floats, produces a diff where a threshold change is one visible line.

import json

def serialise(results) -> str:
    """One JSON object per line, keys sorted, floats rounded — a reviewable diff."""
    rows = []
    for r in sorted(results, key=lambda x: (x["parcel_id"], x["rule_id"])):
        row = {k: (round(v, 3) if isinstance(v, float) else v)
               for k, v in sorted(r.items()) if k != "geometry"}
        rows.append(json.dumps(row, sort_keys=True, default=str))
    return "\n".join(rows) + "\n"

Rounding floats to three decimals is what stops the baseline churning on platform differences. Distances computed by GEOS can differ in the fifteenth decimal place between builds, and an unrounded baseline then fails on a machine where nothing has changed — which trains people to regenerate baselines without reading them, destroying the test’s only real safeguard.

Step 4: Write the test as compare-or-update

The test compares; a flag regenerates. Keeping regeneration behind an explicit flag is what makes the approval step deliberate.

import os
import pathlib
import pytest

BASELINE = pathlib.Path("tests/baselines/rule_pack.jsonl")

def test_rule_pack_against_baseline():
    actual = serialise(evaluate(load_fixtures(), load_rules(), run_date=RUN_DATE))
    if os.environ.get("UPDATE_BASELINES") == "1":
        BASELINE.write_text(actual)
        pytest.skip("baseline updated — review the diff before committing")
    expected = BASELINE.read_text()
    assert actual == expected, "rule pack behaviour changed; review and approve the diff"
Compare by default; regenerate only on purposeThe test compares against the committed baseline; an explicit environment flag regenerates it and skips rather than passing.Evaluate the fixtures at a fixed run dateCRS and fixture count asserted firstSerialise sorted, one record per linefloats rounded so platforms agreeCompare against the committed baselineany difference failsRegeneration skips, and asks for reviewthe approval is the test
A run that regenerated baselines has verified nothing, and reporting it as a pass invites a green build that tested nothing.

The skip rather than pass on update is deliberate: a run that regenerated baselines has not verified anything, and reporting it as a pass invites a green build that tested nothing.

Step 5: Add a mutation test so the suite proves it can fail

A golden suite that has never rejected anything is indistinguishable from one that does not work. Prove it by breaking a rule on purpose and asserting the suite notices.

def test_baseline_detects_a_threshold_change():
    rules = load_rules()
    target = next(r for r in rules if r["id"] == "R1-front-setback-primary")
    target["threshold"] += 1.0          # one foot: the smallest change worth catching

    actual = serialise(evaluate(load_fixtures(), rules, run_date=RUN_DATE))
    assert actual != BASELINE.read_text(), \
        "a one-foot threshold change did not alter any fixture — fixtures are too far " \
        "from the thresholds to discriminate"

This test also validates step 1. If a one-foot change to a real threshold produces no diff, the fixture set contains no parcel near that threshold, and the suite is not testing that rule at all — which is exactly the failure that a passing golden suite otherwise hides.

Step 6: Review baselines like code

The baseline diff is the test’s output, and its value is entirely in someone reading it. A few conventions keep that from decaying:

BASELINE_REVIEW = {
    "commit_separately": "baseline changes in their own commit, never bundled with code",
    "state_the_cause": "the commit message says which rule changed and why",
    "count_the_rows": "an unexpected number of changed rows is the signal to stop",
    "no_bulk_regeneration": "regenerating everything to make CI green destroys the test",
}
The conventions that keep an approved baseline meaningfulFour review habits without which a golden suite degrades into a file people regenerate to make the build green.Baseline changes in their own commitnever bundled with the code changeThe message names the rule and the reasonattribution, while it is still knownAn unexpected row count stops the reviewsomething broader than a threshold changedNo bulk regeneration to reach greenit destroys the only safeguard the test has
The row count is the most useful heuristic: a threshold change should move a handful of fixtures, not all of them.

The row count is the most useful heuristic. A threshold change should move a handful of fixtures; if it moves all thirty-four, something broader changed — a unit, a CRS, an applicability predicate — and that is worth understanding before approving. Attributing a change to its cause is the job of detecting rule drift between pipeline releases, which works at the scale of a full run rather than a fixture set.

Verification

The suite is working when it is deterministic across runs and machines, and when it demonstrably fails on a real change.

# Determinism: two evaluations of the same inputs must be byte-identical.
a = serialise(evaluate(load_fixtures(), load_rules(), run_date=RUN_DATE))
b = serialise(evaluate(load_fixtures(), load_rules(), run_date=RUN_DATE))
assert a == b, "evaluation is not deterministic — check for dict ordering or a clock read"

# Coverage: every rule in the pack must touch at least one fixture.
touched = {r["rule_id"] for r in evaluate(load_fixtures(), load_rules(), RUN_DATE)}
untested = {r["id"] for r in load_rules()} - touched
assert not untested, f"{len(untested)} rules match no fixture: {sorted(untested)[:5]}"

The coverage assertion is the one that keeps the suite honest as the pack grows. A rule pack gains rules faster than a fixture set gains parcels, and without this check the proportion of the pack under test declines quietly with every release.

Common Pitfalls

  • Fixtures made of typical parcels. They are far from every threshold and stay compliant under almost any bug.
  • Unrounded floats in the baseline. Platform-level differences cause spurious failures, which trains people to regenerate without reading.
  • Single-line JSON baselines. The diff is unreadable, so review does not happen, so the test protects nothing.
  • Regenerating to make CI green. This is the failure mode that kills golden suites, and it is why regeneration should skip rather than pass.
  • A clock read inside evaluation. A rule with an effective date compared against today makes the baseline expire on its own.
  • Fixtures from real parcels without redaction. A committed fixture set is published; owner names and addresses in it are a disclosure, not a test artefact.

Frequently Asked Questions

How many fixtures is right?

Enough to cover each criterion in step 1 for each standard family, which usually lands between twenty and sixty parcels. The binding constraint is reviewability: if a typical diff is too long to read carefully, the suite has outgrown its purpose and the extra coverage belongs in a full-run comparison instead.

Should fixtures be real parcels or synthetic?

Both, for different jobs. Real parcels — redacted — catch the messiness that synthetic geometry never has: slivers, concavity, missing attributes, unusual frontage. Synthetic parcels let you place a boundary exactly at a threshold, which is the one thing real data will not do on request. A set of about two-thirds real and one-third constructed works well.

What if a rule change legitimately affects every fixture?

Then approve it and say so in the commit message. A unit change or a CRS change should move everything, and the point of the row count heuristic is not that large diffs are wrong but that they need a stated reason. What is not acceptable is a large diff approved without one.

Does this replace unit tests of the evaluator?

No. Unit tests cover the evaluator’s logic — how a predicate is parsed, how a buffer is built — and golden tests cover the pack’s behaviour. They fail differently and usefully: a unit test says the buffer function is broken, and a golden test says the answers changed. Both are needed.

How does this interact with rule versioning?

Directly: the baseline should record the rule pack’s version alongside the results, so a diff shows both what changed and which version produced it. That version comes from the same mechanism described in versioning rule references in audit trails, and having it in the baseline means a historical determination can be reproduced by checking out the pack the baseline names.

Can the fixture geometry live in the repository?

Yes, and it should — a fixture set held externally is a fixture set that changes without a commit. A few dozen parcels as GeoParquet is well under a megabyte, which is entirely reasonable to version. Store the fixtures and the baseline in the same repository as the rule pack so a single checkout reproduces the test exactly.

Part of: Compliance testing and regression suites