Provenance & Lineage Tracking for Compliance Data

Every compliance verdict a geospatial pipeline emits is only as defensible as the record of how it was produced. Provenance and lineage tracking answers the questions a reviewer, auditor, or opposing counsel will eventually ask: which source datasets fed this result, what transformations reshaped the geometry, which coordinate reference system measured the distances, and exactly which version of the rule declared the parcel compliant. This module sits inside Compliance Reporting & Audit Trail Generation and turns transient pipeline runs into a reconstructable graph, so any historical decision can be replayed byte-for-byte years after the original evaluation.

Prerequisites #

Before wiring provenance capture into a live compliance pipeline, confirm the supporting infrastructure is in place:

• Input layers (parcels, overlays, hydrography) stored with stable, immutable identifiers such as an APN or feature id.
• A content-addressable store or object bucket where raw source files can be preserved by SHA-256 digest.
• geopandas 1.0+, shapely 2.0+, and pyproj 3.4+ installed with GEOS and PROJ correctly linked.
• Rule definitions held in a version-controlled repository so each evaluation can cite a commit SHA or semantic version.
• A serialization target for lineage records, typically JSON on append-only storage or a graph database such as Neo4j.

Provenance metadata is worthless if it is captured but mutable. Treat every record as write-once, and never overwrite a lineage node once the run that produced it has completed.

Core Workflow #

Reliable lineage tracking is a disciplined sequence of capture points rather than a single afterthought log line. The following deterministic steps produce a record that survives audit scrutiny.

1. Fingerprint the inputs. Hash each source file the moment it is read, before any reprojection or repair mutates it. The digest becomes the anchor for everything downstream.
2. Annotate every transform. Each operation records its name, parameters, the CRS in force, and the identifiers of the geometries it consumed and produced.
3. Pin the governing rules. Attach the exact rule version, resolved to an immutable reference, that scored the result.
4. Assemble the graph. Link input nodes to transform nodes to output nodes so lineage can be traversed in either direction.
5. Emit the record immutably. Serialize the assembled provenance and persist it to storage that forbids in-place edits.

The example below captures a source fingerprint and opens a run-scoped provenance record. Note that hashing happens against the file bytes, independent of how geopandas later interprets them.

import hashlib
import json
import uuid
from datetime import datetime, timezone
import geopandas as gpd

def fingerprint_file(path: str) -> str:
    """Return a stable SHA-256 digest of the raw source bytes."""
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(65536), b""):
            h.update(block)
    return h.hexdigest()

def open_provenance(source_path: str) -> dict:
    gdf = gpd.read_file(source_path)
    return {
        "run_id": str(uuid.uuid4()),
        "started_at": datetime.now(timezone.utc).isoformat(),
        "source": {
            "path": source_path,
            "sha256": fingerprint_file(source_path),
            "declared_crs": str(gdf.crs),  # capture CRS as ingested, before any reprojection
            "feature_count": int(len(gdf)),
        },
        "steps": [],
    }

Implementation Patterns #

Two patterns dominate production provenance capture: a decorator that wraps each transform, and an explicit step-recorder threaded through the pipeline. The decorator keeps business logic clean but can obscure which CRS was active; the explicit recorder is more verbose but gives auditors an unambiguous trail. For compliance work, prefer the explicit form and always record the CRS at the point of measurement.

The house rule is non-negotiable: project to a linear, metric CRS before any distance, area, or buffer operation, and log that CRS in the same step that performed the measurement. The recorder below captures a reprojection and a buffer as two distinct, linked steps.

def record_step(prov: dict, op: str, params: dict, crs: str, inputs: list[str]) -> str:
    """Append one immutable step node and return its id for downstream linkage."""
    step_id = f"step-{len(prov['steps']):03d}"
    prov["steps"].append({
        "step_id": step_id,
        "operation": op,
        "params": params,
        "crs": crs,           # the CRS in force for THIS operation
        "inputs": inputs,     # ids of upstream nodes consumed
    })
    return step_id

def buffer_with_lineage(prov, gdf, distance_m, metric_crs="EPSG:26915"):
    src_id = prov["source"]["sha256"]
    projected = gdf.to_crs(metric_crs)  # project BEFORE measuring
    reproj_id = record_step(prov, "to_crs", {"target": metric_crs}, metric_crs, [src_id])
    projected["geometry"] = projected.geometry.buffer(distance_m)
    record_step(prov, "buffer", {"distance_m": distance_m}, metric_crs, [reproj_id])
    return projected

Building the lineage graph itself is a matter of treating each recorded inputs list as directed edges. Once assembled, the graph doubles as the backbone of a validation log; the companion module on validation log design consumes these same step nodes to emit per-run structured logs without duplicating capture logic.

Edge Cases & Geometry Repair #

Provenance capture must survive the messy geometries that legacy municipal data routinely produces. Three cases recur:

• Repaired geometries drift from their fingerprint. When make_valid rewrites a self-intersecting polygon, the output no longer matches the hashed input. Record the repair as its own step so the divergence is explicit rather than silent.
• Null and empty geometries. Dropping invalid records changes the feature count. Log the count before and after so a reviewer can reconcile the difference.
• CRS with no declared authority. A layer whose crs is None cannot be trusted for measurement. Capture the missing CRS as a provenance warning and refuse to buffer until it is assigned.

from shapely import make_valid

def repair_with_lineage(prov, gdf, crs):
    before = len(gdf)
    gdf = gdf.dropna(subset=["geometry"]).copy()
    invalid_mask = ~gdf.geometry.is_valid
    gdf.loc[invalid_mask, "geometry"] = gdf.loc[invalid_mask, "geometry"].apply(make_valid)
    record_step(prov, "repair", {
        "dropped": before - len(gdf),
        "repaired": int(invalid_mask.sum()),
    }, crs, [prov["source"]["sha256"]])
    return gdf

Audit Logging & Provenance #

The finished record must name the input layers by digest, list every transform with its parameters, state each CRS used, and cite the versioned rule references that produced the verdict. Serializing to JSON on append-only storage keeps the record tamper-evident and human-readable. Capturing the exact rule reference is important enough to warrant its own guide; see versioning rule references in audit trails for pinning strategies that tie each evaluation to an immutable commit SHA or semantic version.

def close_provenance(prov: dict, rule_ref: dict, path: str) -> None:
    prov["rule_reference"] = rule_ref          # e.g. {"repo": "zoning-rules", "sha": "9f2c1a7"}
    prov["finished_at"] = datetime.now(timezone.utc).isoformat()
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(prov, fh, indent=2, sort_keys=True)  # deterministic key order aids diffing

Because the mechanics of hashing and recording each pipeline stage are involved, the dedicated walkthrough on tracking data lineage across geospatial ETL steps expands this pattern into a full extract-transform-load example.

Troubleshooting #

• Fingerprints change between runs on identical data. Shapefiles bundle sidecar files; hash a canonical serialization such as GeoPackage or the raw component bytes, not an in-memory GeoDataFrame whose row order can vary.
• Lineage graph has orphan nodes. A transform recorded inputs that reference an id never emitted upstream. Validate that every inputs entry resolves to an existing node before serializing.
• Non-reproducible results despite pinned rules. An implicit CRS default likely differs across environments. Assert the metric CRS explicitly in each measurement step rather than relying on library defaults.
• Provenance file balloons in size. Storing full geometries inline is wasteful; store geometry references or digests and keep the heavy features in the content-addressable store. The PyProj CRS documentation helps you record compact, authoritative CRS identifiers instead of verbose WKT.

Conclusion #

Provenance and lineage tracking converts a compliance pipeline from a black box into a transparent, replayable system of record. By fingerprinting inputs, annotating every transform with its CRS, pinning rule versions, and serializing the assembled graph to immutable storage, teams gain the reproducibility that regulators and courts demand. The broader reporting and audit trail area depends on this foundation: no certificate or dashboard is trustworthy unless the lineage behind it can be reconstructed exactly, from raw source bytes to final verdict.