Detecting Rule Drift Between Pipeline Releases
A golden-file suite catches what changed on thirty parcels. A release changes verdicts on a county, and the question that gates the merge is different: which parcels moved, and why. The second half is the hard part, because between two releases the rule pack changed, the parcel fabric was refreshed, and GEOS was upgraded — and a verdict that moved could be attributable to any of them. This guide compares two full runs, classifies every changed verdict by cause, and produces the report a release decision is actually made from. It is the regression run that gates a merge in compliance testing and regression suites.
Prerequisites
Step-by-step
Step 1: Diff the verdicts, keyed on parcel and rule
The comparison is an outer join on the natural key. An outer join rather than an inner one is essential: parcels and rules that appear in only one run are drift too, and an inner join hides exactly those.
import pandas as pd
KEY = ["parcel_id", "rule_id"]
def diff_runs(before: pd.DataFrame, after: pd.DataFrame) -> pd.DataFrame:
m = before.merge(after, on=KEY, how="outer", suffixes=("_before", "_after"),
indicator=True)
m["change"] = "unchanged"
m.loc[m["_merge"] == "left_only", "change"] = "disappeared"
m.loc[m["_merge"] == "right_only", "change"] = "appeared"
both = m["_merge"] == "both"
m.loc[both & (m["verdict_before"] != m["verdict_after"]), "change"] = "verdict_moved"
m.loc[both & (m["verdict_before"] == m["verdict_after"])
& ((m["measured_before"] - m["measured_after"]).abs() > 1e-6),
"change"] = "measurement_moved"
return mmeasurement_moved — same verdict, different number — matters more than it appears. It is the early warning: a measurement that shifted without crossing a threshold this time will cross one on the next refresh, and it is the signature of a geometry or CRS change that has not yet become visible.
Step 2: Compare the manifests to enumerate the candidate causes
Before attributing anything, establish what actually differs between the runs. Everything that changed is a suspect; everything that did not is eliminated.
def manifest_delta(before: dict, after: dict) -> dict:
return {
"rules_changed": before["rule_pack_sha256"] != after["rule_pack_sha256"],
"parcels_changed": before["parcels_sha256"] != after["parcels_sha256"],
"districts_changed": before["districts_sha256"] != after["districts_sha256"],
"code_changed": before["pipeline_version"] != after["pipeline_version"],
"libs_changed": {k: (before["libs"].get(k), v)
for k, v in after["libs"].items()
if before["libs"].get(k) != v},
}If exactly one thing changed, attribution is done. That is the argument for releasing changes one axis at a time where possible — a release that bundles a rule edit with a data refresh cannot be attributed without the extra run in step 3.
Step 3: Attribute by holding one axis fixed
Where more than one thing changed, the way to separate them is a third run: the new code and rules against the old data. The differences it shows are attributable to code and rules; whatever remains between it and the full new run is attributable to data.
def attribute(before, after, control):
"""control = new code + new rules, old data. Three-way attribution."""
code_effect = diff_runs(before, control)
data_effect = diff_runs(control, after)
moved_by_code = set(code_effect.loc[code_effect["change"] != "unchanged"]
.set_index(KEY).index)
moved_by_data = set(data_effect.loc[data_effect["change"] != "unchanged"]
.set_index(KEY).index)
return {
"code_or_rules_only": moved_by_code - moved_by_data,
"data_only": moved_by_data - moved_by_code,
"both": moved_by_code & moved_by_data,
}The control run costs one extra execution and removes the argument that otherwise consumes a release review. It is worth automating rather than performing when the diff looks alarming, because the runs where nobody thought to do it are the runs where the attribution mattered.
Step 4: Attribute the rule-caused changes to specific rules
Within the code-and-rules bucket, a diff of the packs themselves narrows it further, and usually to a single rule.
def rule_pack_diff(before_rules, after_rules):
b = {r["id"]: r for r in before_rules}
a = {r["id"]: r for r in after_rules}
changed = {}
for rid in b.keys() & a.keys():
fields = {k for k in (b[rid].keys() | a[rid].keys())
if b[rid].get(k) != a[rid].get(k)}
if fields:
changed[rid] = {f: (b[rid].get(f), a[rid].get(f)) for f in sorted(fields)}
return {"added": sorted(a.keys() - b.keys()),
"removed": sorted(b.keys() - a.keys()),
"modified": changed}Cross-referencing this against the changed verdicts closes the loop: a rule whose threshold moved should account for changes on exactly the parcels it applies to, and a changed verdict on a parcel no modified rule touches is unexplained — which is the finding that should block a release.
Step 5: Quantify the drift in terms a reviewer can act on
Counts of changed rows are not a release decision. Direction and materiality are.
def drift_summary(diff: pd.DataFrame) -> dict:
moved = diff[diff["change"] == "verdict_moved"]
to_violation = moved[(moved["verdict_before"] == "complies")
& (moved["verdict_after"] == "exceeds")]
to_compliant = moved[(moved["verdict_before"] == "exceeds")
& (moved["verdict_after"] == "complies")]
return {
"total_rows": len(diff),
"verdicts_moved": len(moved),
"moved_pct": 100.0 * len(moved) / max(len(diff), 1),
"newly_violating": len(to_violation),
"newly_compliant": len(to_compliant),
"appeared": int((diff["change"] == "appeared").sum()),
"disappeared": int((diff["change"] == "disappeared").sum()),
"measurement_only": int((diff["change"] == "measurement_moved").sum()),
}newly_compliant deserves more scrutiny than newly_violating, which is the opposite of most people’s instinct. A parcel that becomes non-compliant gets reviewed by whoever receives the finding; a parcel that quietly becomes compliant is a finding that stopped being reported, and nobody is looking for it.
Step 6: Gate the release on thresholds that were agreed in advance
The gate has to be a rule rather than a judgement made while looking at the number, or it becomes a negotiation.
DRIFT_GATE = {
"max_moved_pct": 1.0, # above this, the release needs a written explanation
"max_unexplained": 0, # a change no modified rule accounts for blocks
"max_disappeared": 0, # a parcel losing a verdict is always a defect
}
def gate(summary, unexplained_count):
failures = []
if summary["moved_pct"] > DRIFT_GATE["max_moved_pct"]:
failures.append(f"{summary['moved_pct']:.2f}% of verdicts moved")
if unexplained_count > DRIFT_GATE["max_unexplained"]:
failures.append(f"{unexplained_count} changes attributable to nothing")
if summary["disappeared"] > DRIFT_GATE["max_disappeared"]:
failures.append(f"{summary['disappeared']} parcel-rule pairs lost a verdict")
return failuresmax_unexplained: 0 is the one worth holding firm on. A drift percentage above a threshold is often legitimate and explainable; a verdict that changed with no rule change and no data change to account for it is a bug, and it is the kind that gets discovered later by an applicant.
Verification
The comparison itself needs testing — a diff that misses changes is worse than none, because it certifies a release it did not check.
# A run compared against itself must show nothing.
self_diff = diff_runs(run_a, run_a.copy())
assert (self_diff["change"] == "unchanged").all(), "diff reports spurious changes"
# An injected change must be found, with the right classification.
mutated = run_a.copy()
mutated.loc[mutated.index[0], "verdict"] = "exceeds"
d = diff_runs(run_a, mutated)
assert (d["change"] == "verdict_moved").sum() == 1
# A dropped row must be found too.
d2 = diff_runs(run_a, run_a.iloc[1:])
assert (d2["change"] == "disappeared").sum() == 1
# The key must actually be unique, or the merge silently multiplies rows.
assert not run_a.duplicated(subset=KEY).any(), "duplicate keys — the diff is invalid"That last assertion prevents the failure that makes every other number meaningless. A duplicated parcel-rule key turns the outer join into a partial cross product, inflating the row count and the drift percentage together, so the summary looks alarming for a reason that has nothing to do with the release.
Common Pitfalls
- Inner-joining the two runs. Parcels present in only one run are drift, and an inner join is precisely blind to them.
- Comparing summaries rather than rows. Two runs with identical counts can differ on thousands of parcels, in offsetting directions.
- Bundling a rule change with a data refresh. Attribution then requires the control run, and without it the review becomes an argument.
- Ignoring
newly_compliant. A finding that stopped being reported has no one watching for it. - Ignoring measurement moves that did not cross a threshold. They are the leading indicator of the next release’s verdict changes.
- Setting the drift gate after seeing the number. A threshold chosen to accommodate the current release is not a gate.
Frequently Asked Questions
How much drift is normal?
Between releases that change no rules and refresh no data, zero — and anything else is a determinism bug worth chasing. With a quarterly parcel refresh, a fraction of a percent is typical and reflects real fabric edits. With a rule change, the expected number is whatever that rule’s applicability count implies, which is computable in advance and worth predicting before running the comparison.
Should the reference run’s full output really be kept?
Yes. Storing verdicts for a county is a few tens of megabytes compressed — trivial against the cost of not being able to compare. Keeping only summaries means the first genuinely confusing release is the one where the comparison cannot be made.
What about non-determinism inside the pipeline?
It has to be eliminated before drift detection is meaningful, because non-determinism looks exactly like drift. The usual sources are dictionary or set iteration order leaking into tie-breaks, parallel workers writing in completion order, and a clock read inside evaluation. The self-comparison in the verification section is what catches all three.
How does this relate to golden-file tests?
They are the same technique at different scales and they catch different things. Golden-file tests run in seconds on every commit and catch edge-case regressions; a full drift comparison runs per release and catches effects that only appear at scale — an applicability predicate that matches far more parcels than intended, for instance, which no fixture set would reveal.
Can drift be attributed to a library upgrade specifically?
Yes, using the same control-run technique with the library pinned instead of the data. GEOS upgrades are the usual culprit and they show up characteristically: tiny measurement changes on a large number of parcels, with verdicts moving only where a value sat almost exactly on a threshold. That signature is distinctive enough to recognise once you have seen it.
Where should the drift report go?
Into the release record, alongside the manifests, and into the pull request. It is the artefact that says what a release did, and it is the first thing anyone will want when a determination is questioned months later — which is the same reason tracking data lineage across geospatial ETL steps keeps manifests rather than summaries.
Related
Part of: Compliance testing and regression suites
- Writing golden-file tests for zoning rule packs — the same idea at fixture scale.
- Tracking data lineage across geospatial ETL steps — the manifests attribution depends on.
- Versioning rule references in audit trails — identifying which pack produced a verdict.
- Chunking county-scale runs by spatial tile — stable partitions that localise a diff.
- Structured JSON logging for geospatial pipelines — the run records this reads.