Provenance & Lineage Tracking for Compliance Data

Every compliance verdict a geospatial pipeline emits is only as defensible as the record of how it was produced. Provenance and lineage tracking answers the questions a reviewer, auditor, or opposing counsel will eventually ask: which source datasets fed this result, what transformations reshaped the geometry, which coordinate reference system measured the distances, and exactly which version of the rule declared the parcel compliant. This module sits inside Compliance Reporting & Audit Trail Generation and turns transient pipeline runs into a reconstructable graph, so any historical decision can be replayed byte-for-byte years after the original evaluation.

Prerequisites #

Before wiring provenance capture into a live compliance pipeline, confirm the supporting infrastructure is in place:

• Input layers (parcels, overlays, hydrography) stored with stable, immutable identifiers such as an APN or feature id.
• A content-addressable store or object bucket where raw source files can be preserved by SHA-256 digest.
• geopandas 1.0+, shapely 2.0+, and pyproj 3.4+ installed with GEOS and PROJ correctly linked.
• Rule definitions held in a version-controlled repository so each evaluation can cite a commit SHA or semantic version.
• A serialization target for lineage records, typically JSON on append-only storage or a graph database such as Neo4j.

Provenance metadata is worthless if it is captured but mutable. Treat every record as write-once, and never overwrite a lineage node once the run that produced it has completed.

Core Workflow #

Reliable lineage tracking is a disciplined sequence of capture points rather than a single afterthought log line. The following deterministic steps produce a record that survives audit scrutiny.

1. Fingerprint the inputs. Hash each source file the moment it is read, before any reprojection or repair mutates it. The digest becomes the anchor for everything downstream.
2. Annotate every transform. Each operation records its name, parameters, the CRS in force, and the identifiers of the geometries it consumed and produced.
3. Pin the governing rules. Attach the exact rule version, resolved to an immutable reference, that scored the result.
4. Assemble the graph. Link input nodes to transform nodes to output nodes so lineage can be traversed in either direction.
5. Emit the record immutably. Serialize the assembled provenance and persist it to storage that forbids in-place edits.

The example below captures a source fingerprint and opens a run-scoped provenance record. Note that hashing happens against the file bytes, independent of how geopandas later interprets them.

import hashlib
import json
import uuid
from datetime import datetime, timezone
import geopandas as gpd

def fingerprint_file(path: str) -> str:
    """Return a stable SHA-256 digest of the raw source bytes."""
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(65536), b""):
            h.update(block)
    return h.hexdigest()

def open_provenance(source_path: str) -> dict:
    gdf = gpd.read_file(source_path)
    return {
        "run_id": str(uuid.uuid4()),
        "started_at": datetime.now(timezone.utc).isoformat(),
        "source": {
            "path": source_path,
            "sha256": fingerprint_file(source_path),
            "declared_crs": str(gdf.crs),  # capture CRS as ingested, before any reprojection
            "feature_count": int(len(gdf)),
        },
        "steps": [],
    }

Implementation Patterns #

Two patterns dominate production provenance capture: a decorator that wraps each transform, and an explicit step-recorder threaded through the pipeline. The decorator keeps business logic clean but can obscure which CRS was active; the explicit recorder is more verbose but gives auditors an unambiguous trail. For compliance work, prefer the explicit form and always record the CRS at the point of measurement.

The house rule is non-negotiable: project to a linear, metric CRS before any distance, area, or buffer operation, and log that CRS in the same step that performed the measurement. The recorder below captures a reprojection and a buffer as two distinct, linked steps.

def record_step(prov: dict, op: str, params: dict, crs: str, inputs: list[str]) -> str:
    """Append one immutable step node and return its id for downstream linkage."""
    step_id = f"step-{len(prov['steps']):03d}"
    prov["steps"].append({
        "step_id": step_id,
        "operation": op,
        "params": params,
        "crs": crs,           # the CRS in force for THIS operation
        "inputs": inputs,     # ids of upstream nodes consumed
    })
    return step_id

def buffer_with_lineage(prov, gdf, distance_m, metric_crs="EPSG:26915"):
    src_id = prov["source"]["sha256"]
    projected = gdf.to_crs(metric_crs)  # project BEFORE measuring
    reproj_id = record_step(prov, "to_crs", {"target": metric_crs}, metric_crs, [src_id])
    projected["geometry"] = projected.geometry.buffer(distance_m)
    record_step(prov, "buffer", {"distance_m": distance_m}, metric_crs, [reproj_id])
    return projected

Building the lineage graph itself is a matter of treating each recorded inputs list as directed edges. Once assembled, the graph doubles as the backbone of a validation log; the companion module on validation log design consumes these same step nodes to emit per-run structured logs without duplicating capture logic.

Edge Cases & Geometry Repair #

Provenance capture must survive the messy geometries that legacy municipal data routinely produces. Three cases recur:

• Repaired geometries drift from their fingerprint. When make_valid rewrites a self-intersecting polygon, the output no longer matches the hashed input. Record the repair as its own step so the divergence is explicit rather than silent.
• Null and empty geometries. Dropping invalid records changes the feature count. Log the count before and after so a reviewer can reconcile the difference.
• CRS with no declared authority. A layer whose crs is None cannot be trusted for measurement. Capture the missing CRS as a provenance warning and refuse to buffer until it is assigned.

from shapely import make_valid

def repair_with_lineage(prov, gdf, crs):
    before = len(gdf)
    gdf = gdf.dropna(subset=["geometry"]).copy()
    invalid_mask = ~gdf.geometry.is_valid
    gdf.loc[invalid_mask, "geometry"] = gdf.loc[invalid_mask, "geometry"].apply(make_valid)
    record_step(prov, "repair", {
        "dropped": before - len(gdf),
        "repaired": int(invalid_mask.sum()),
    }, crs, [prov["source"]["sha256"]])
    return gdf

Audit Logging & Provenance #

The finished record must name the input layers by digest, list every transform with its parameters, state each CRS used, and cite the versioned rule references that produced the verdict. Serializing to JSON on append-only storage keeps the record tamper-evident and human-readable. Capturing the exact rule reference is important enough to warrant its own guide; see versioning rule references in audit trails for pinning strategies that tie each evaluation to an immutable commit SHA or semantic version.

def close_provenance(prov: dict, rule_ref: dict, path: str) -> None:
    prov["rule_reference"] = rule_ref          # e.g. {"repo": "zoning-rules", "sha": "9f2c1a7"}
    prov["finished_at"] = datetime.now(timezone.utc).isoformat()
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(prov, fh, indent=2, sort_keys=True)  # deterministic key order aids diffing

Because the mechanics of hashing and recording each pipeline stage are involved, the dedicated walkthrough on tracking data lineage across geospatial ETL steps expands this pattern into a full extract-transform-load example.

Troubleshooting #

• Fingerprints change between runs on identical data. Shapefiles bundle sidecar files; hash a canonical serialization such as GeoPackage or the raw component bytes, not an in-memory GeoDataFrame whose row order can vary.
• Lineage graph has orphan nodes. A transform recorded inputs that reference an id never emitted upstream. Validate that every inputs entry resolves to an existing node before serializing.
• Non-reproducible results despite pinned rules. An implicit CRS default likely differs across environments. Assert the metric CRS explicitly in each measurement step rather than relying on library defaults.
• Provenance file balloons in size. Storing full geometries inline is wasteful; store geometry references or digests and keep the heavy features in the content-addressable store. The PyProj CRS documentation helps you record compact, authoritative CRS identifiers instead of verbose WKT.

Provenance is also what makes a correction proportionate. When a supplier reissues a layer with a fix, the reverse index turns “we may need to re-evaluate everything” into a specific list of affected verdicts, which is usually a small fraction and occasionally none at all. Without it, the safe response to any data correction is a full re-run, which is expensive enough that teams start avoiding corrections.

That proportionality argument is worth making explicitly to whoever is nervous about the cost of provenance: recording it well is what keeps corrections cheap.

Software Provenance Counts Too #

Data lineage gets the attention, and software lineage causes at least as many unexplained differences between runs. A compliance result depends on GEOS, PROJ, GDAL, the transformation grids installed, and the Python libraries wrapping them, and every one of those changes on its own schedule.

The failure this produces is characteristic: a run repeated months later gives slightly different areas, or a geometry repair produces a different result, and there is no data change to blame. Without recorded software versions the investigation is a guess; with them it is a diff.

What to record is short and worth being complete about — the versions of GEOS, PROJ and GDAL, the Python geospatial libraries, and the identity of the container image if there is one. Recorded together in the run manifest, they turn “why did this change?” into a question with an answer.

def software_provenance() -> dict:
    """Versions that can change a geometric result without any data changing."""
    import shapely, pyproj, geopandas, fiona
    return {
        "geos": shapely.geos_version_string,
        "proj": pyproj.proj_version_str,
        "gdal": fiona.__gdal_version__,
        "shapely": shapely.__version__,
        "pyproj": pyproj.__version__,
        "geopandas": geopandas.__version__,
        "image": os.environ.get("IMAGE_DIGEST", "unpinned"),
    }

The unpinned fallback in that snippet is deliberate rather than defensive. A run that cannot name its image is a run whose environment cannot be reconstructed, and recording that fact honestly is more useful than omitting the field — it tells a later investigator exactly why the difference cannot be explained.

Pinning matters as much as recording. An environment specified by a version range will drift between runs without anybody choosing it, and the drift shows up first in the last decimal place of an area and later in a verdict at a threshold. Pin the image, record the digest, and upgrade deliberately with a regression run to attribute whatever moves.

Recording Provenance Without Slowing the Run #

Provenance capture is often resisted on performance grounds, and the resistance is usually aimed at an implementation nobody should have written. Recording a hash per parcel, or writing a lineage record inside the evaluation loop, does slow a run measurably. Recording at the right granularity does not.

The right granularity is the layer, not the feature. A county parcel fabric has one content hash, computed once when the snapshot is taken, and every verdict derived from it references that one hash. Hashing the bytes of a few gigabyte-scale layers at ingest costs seconds; hashing a million geometries individually costs minutes and answers no question the layer hash does not.

The same applies to transformation records. A run applies the same reprojection to a whole layer, so the transformation is a property of the run, recorded once in the manifest, rather than a property of each feature. Only genuinely per-feature events — a geometry repaired, a parcel quarantined, a split recorded — deserve per-feature records, and those are rare by construction.

def snapshot_layer(path, store):
    """Hash once, at read time, before anything transforms the bytes."""
    digest = sha256_file(path)
    store.put(f"snapshots/{digest}", path)
    return {"layer": Path(path).stem, "sha256": digest,
            "retrieved_at": utc_now_iso(), "bytes": Path(path).stat().st_size}

With that structure, provenance adds a fixed cost per run and a constant number of bytes per verdict, and the argument about overhead disappears. What remains is the discipline of doing it at ingest rather than afterwards, because a hash computed after reprojection describes something the supplier never published and cannot be compared against anything.

Recording provenance well is therefore less about volume than about placement: a few facts, captured at the right moment, in a form that later queries can use.

Identity Is the Hard Part #

Lineage tracking is usually described as recording transformations, and the transformations are the easy half. The hard half is identity: knowing that the parcel in today’s run is the same parcel as in last year’s, or knowing precisely which version of a layer a verdict was computed from.

Three identifiers are needed and they are frequently conflated. A dataset identity names the layer and its edition — “county parcel fabric, published 2026-05-02” — and is what a human refers to. A content identity is a hash of the bytes, and it is what proves two runs read the same thing; editions can be republished silently, hashes cannot. A feature identity names an individual parcel across time, and it is the one most likely to be missing, because export processes routinely renumber rows.

Where a stable published identifier exists — an assessor’s parcel number is the usual candidate — use it and record it. Where it does not, a content-derived identity over the normalised geometry is a workable substitute that survives attribute edits. What does not work is an export-generated row id, which is stable only until the next export.

Splits and merges break feature identity by design, and they need to be recorded as lineage events rather than treated as an identifier problem. A parcel that becomes two has ended, and two parcels have begun with a stated parent; the compliance history attached to the parent is then explicitly carried forward or closed, rather than silently orphaned.

Lineage That Answers Real Questions #

A lineage graph is only worth building if it answers questions people actually ask. In practice there are four, and designing backwards from them keeps the model small.

“Which inputs produced this verdict?” — the common one, asked whenever a result is disputed. It needs a verdict record carrying the content hashes of every layer that contributed, which is a flat list rather than a graph.

“Which verdicts are affected by this corrected layer?” — asked after a data fix. It needs the reverse index: given a content hash, find every verdict that referenced it. This is the query that decides how much re-evaluation a correction triggers.

“What changed between these two runs?” — asked when a verdict flips. It needs the run manifests side by side: rule version, layer editions, configuration, software versions. The differing field is usually the answer.

“Where did this parcel come from?” — asked about subdivisions and boundary corrections. It needs the parent-child lineage events described above.

None of these requires a general-purpose lineage system, and adopting one before the questions are clear usually produces an elaborate graph that answers none of them well. A verdict record carrying input hashes, a run manifest, and a small table of feature lineage events cover all four, and each is a few lines of code to write and cheap to query.

def verdicts_affected_by(conn, layer_hash: str):
    """Reverse lineage: everything computed from a layer now known to be wrong."""
    return conn.execute(
        "SELECT parcel_id, rule_id, rule_version, run_id "
        "FROM verdicts WHERE %s = ANY(input_hashes) "
        "ORDER BY run_id, parcel_id",
        (layer_hash,),
    ).fetchall()

Provenance work rarely feels urgent while a pipeline is being built and is the first thing asked for when one of its results is questioned, which is the whole argument for doing it at the start.

Related #

Part of: Compliance reporting and audit trail generation

• Tracking data lineage across geospatial ETL steps — the transformation half, in code.
• Versioning rule references in audit trails — the rule side of the same problem.
• Validation log design — the event stream lineage is derived from.
• Zoning layer ingestion strategies — where dataset and content identity are first established.

Conclusion #

Provenance and lineage tracking converts a compliance pipeline from a black box into a transparent, replayable system of record. By fingerprinting inputs, annotating every transform with its CRS, pinning rule versions, and serializing the assembled graph to immutable storage, teams gain the reproducibility that regulators and courts demand. The broader reporting and audit trail area depends on this foundation: no certificate or dashboard is trustworthy unless the lineage behind it can be reconstructed exactly, from raw source bytes to final verdict.