Redacting Sensitive Parcel Data from Logs

Parcel records carry owner names, mailing addresses and occasionally more, and a validation log that quotes a failing record quotes all of it. Logs are the most widely shared artefact a pipeline produces — they go to aggregation services, into support tickets, into screenshots — so a log that echoes parcel attributes is a disclosure channel nobody designed. This guide redacts at the point of emission rather than downstream, preserves the ability to reconstruct a run through stable pseudonyms, and includes the test that proves the redaction actually fires. It is what must not travel with the logs described in validation log design.

Prerequisites #

• A structured logging setup, per structured JSON logging for geospatial pipelines
• A classification of the parcel schema’s fields — which are public record, which are not
• A per-deployment secret for pseudonym derivation, stored outside the repository
• Knowledge of the local rules on parcel data, which vary considerably by jurisdiction
• Access to whoever decides what may leave the organisation, since this is their call

Step-by-step #

Step 1: Classify the schema before writing any filter #

Redaction driven by pattern matching over field values is unreliable in both directions. Classify the schema instead — it is a finite list and it is decided once.

FIELD_POLICY = {
    # Identifiers safe to log: they are the point of the log.
    "parcel_id": "public",
    "zoning_district": "public",
    "rule_id": "public",
    "lot_area": "public",

    # Public record in most jurisdictions, but not appropriate for shared telemetry.
    "owner_name": "pseudonymise",
    "situs_address": "pseudonymise",
    "mailing_address": "pseudonymise",

    # Never emitted in any form.
    "owner_phone": "drop",
    "owner_email": "drop",
    "api_token": "drop",
    "connection_string": "drop",
}

The three-way split matters. Dropping everything sensitive makes logs useless for debugging — you cannot tell whether two errors concern the same owner. Pseudonymising preserves the correlation that debugging needs while removing the identity, and it is the right treatment for most of what falls in the middle.

Step 2: Derive stable pseudonyms with a keyed hash #

A pseudonym must be stable across a run and across runs — so the same owner is the same token everywhere — and must not be reversible by anyone holding the log. A keyed hash gives both; an unkeyed one does not, since the space of names and addresses is small enough to enumerate.

import hashlib
import hmac
import os

_PEPPER = os.environ["LOG_PSEUDONYM_KEY"].encode()   # absent -> the pipeline will not start

def pseudonym(value: str, prefix: str = "own") -> str:
    """Stable within a deployment, unlinkable without the key."""
    if value is None:
        return None
    digest = hmac.new(_PEPPER, str(value).strip().lower().encode(),
                      hashlib.sha256).hexdigest()
    return f"{prefix}_{digest[:12]}"

Reading the key with os.environ[...] rather than .get() is deliberate: a missing key should stop the pipeline, not silently fall back to an unkeyed hash. That fallback is the mechanism by which redaction is quietly disabled in exactly the environment where it was misconfigured.

Step 3: Redact in the log filter, not at the call sites #

Redaction applied at each logging call is redaction that will be forgotten at the next one. Put it in the pipeline that every record passes through.

import logging

DROP = "[redacted]"

class RedactingFilter(logging.Filter):
    def filter(self, record):
        extra = getattr(record, "context", None)
        if isinstance(extra, dict):
            record.context = self._clean(extra)
        record.msg = _clean_text(str(record.msg))   # defined in Step 4
        return True

    def _clean(self, obj):
        if isinstance(obj, dict):
            out = {}
            for k, v in obj.items():
                policy = FIELD_POLICY.get(k, "unknown")
                if policy == "drop":
                    out[k] = DROP
                elif policy == "pseudonymise":
                    out[k] = pseudonym(v, prefix=k[:3])
                elif policy == "unknown":
                    out[k] = DROP          # fail closed on fields nobody classified
                else:
                    out[k] = self._clean(v)
            return out
        if isinstance(obj, (list, tuple)):
            return [self._clean(v) for v in obj]
        return obj

Failing closed on unclassified fields is the design decision that makes this hold up over time. A schema gains columns, and a filter that passes anything it does not recognise leaks every new column by default — whereas one that redacts them produces a visible [redacted] that someone classifies.

Step 4: Handle the free-text case, which the field filter cannot #

Exception messages and formatted strings carry values that never passed through a classified field — a KeyError naming an address, a database error quoting a connection string.

import re

PATTERNS = [
    (re.compile(r"postgres(?:ql)?://[^\s\"']+"), "postgresql://[redacted]"),
    (re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b"), "[email redacted]"),
    (re.compile(r"\btoken=[\w.\-]+"), "token=[redacted]"),
    (re.compile(r"\b\d{1,5}\s+[A-Z][a-z]+\s+(?:St|Ave|Rd|Blvd|Ln|Dr|Ct)\b"), "[address]"),
]

def _clean_text(text: str) -> str:
    for pattern, replacement in PATTERNS:
        text = pattern.sub(replacement, text)
    return text

Pattern matching over free text is a backstop and should be described as one. It will miss addresses in unusual formats and occasionally redact something innocuous. The primary defence is not logging free text that contains record values at all — log the parcel identifier and the field name, not the field’s contents.

Step 5: Keep the mapping, separately and access-controlled #

Debugging sometimes requires resolving a pseudonym. That is legitimate, and the way to support it is a mapping stored where the logs are not.

import pandas as pd

def record_mapping(values, prefix, out_path):
    """Written to controlled storage; never to the log destination."""
    rows = [{"pseudonym": pseudonym(v, prefix), "value": v} for v in sorted(set(values))]
    pd.DataFrame(rows).to_parquet(out_path, index=False)

The mapping is the sensitive artefact, and treating it as such — separate storage, separate access, its own retention period — is what makes the pseudonyms in the log genuinely safe to share. A mapping stored alongside the logs converts pseudonymisation back into plain disclosure with extra steps.

Step 6: Apply the same policy to every artefact that leaves #

Logs are the obvious channel and not the only one. Test fixtures, error reports, cached intermediate files and support exports all carry the same fields.

EXPORT_POLICY = {
    "validation_log": "redact",
    "test_fixtures": "redact",        # committed to a repository — treat as published
    "error_reports": "redact",
    "compliance_certificate": "full", # the determination names the parcel by design
    "internal_working_store": "full",
}

The certificate is deliberately exempt, and the distinction is worth stating: a determination is about a specific parcel and must identify it, while a log is about a pipeline’s behaviour and does not need to. Committed test fixtures are the case most often missed — a fixture set is published the moment it is pushed, which is why golden-file tests should be built from redacted parcels.

Verification #

Redaction that has never been tested is a configuration setting. Feed it the things it must refuse.

import io
import json

def test_redaction_fires():
    buf = io.StringIO()
    handler = logging.StreamHandler(buf)
    handler.addFilter(RedactingFilter())
    log = logging.getLogger("test"); log.addHandler(handler); log.setLevel(logging.INFO)

    log.info("check failed", extra={"context": {
        "parcel_id": "0123-456-789",
        "owner_name": "Jane Q. Public",
        "situs_address": "42 Elm St",
        "api_token": "sk-live-abcdef123456",
        "unclassified_new_column": "whatever this is",
    }})
    out = buf.getvalue()

    assert "0123-456-789" in out, "the parcel id must survive — the log is useless without it"
    for secret in ("Jane Q. Public", "42 Elm St", "sk-live-abcdef123456", "whatever this is"):
        assert secret not in out, f"{secret!r} reached the log"
    assert "own_" in out, "owner was dropped rather than pseudonymised"

def test_pseudonyms_are_stable_and_distinct():
    assert pseudonym("Jane Q. Public") == pseudonym("jane q. public  ")
    assert pseudonym("Jane Q. Public") != pseudonym("John Q. Public")

The first assertion is as important as the rest. Redaction that removes the parcel identifier makes the log unusable, which guarantees somebody turns it off — so the test asserts both that the secrets left and that the useful content stayed.

Common Pitfalls #

• Redacting downstream. A filter in the aggregation service does not protect the file on disk, the terminal, or the screenshot in a ticket.
• Passing unclassified fields through. The schema gains columns, and a permissive default leaks each new one until someone notices.
• Unkeyed hashes as pseudonyms. Names and addresses are enumerable, so an unkeyed digest is reversible by anyone who wants to.
• Storing the mapping with the logs. It converts pseudonymisation back into disclosure.
• Relying on regular expressions as the primary defence. They are a backstop for free text; the field policy is the mechanism.
• Logging whole records on error. log.error("failed", extra={"row": row.to_dict()}) defeats every field policy at once.
• Committing unredacted fixtures. A repository is a publication channel, and history keeps what was removed.

Frequently Asked Questions #

Parcel ownership is public record — why redact it? #

Because public record and freely redistributable are different things. Most jurisdictions publish ownership through a controlled channel with terms attached, and a log shipped to a third-party aggregation service is a bulk export nobody authorised. The pipeline also does not need the data: a parcel identifier resolves to an owner for anyone entitled to look, which is exactly the property that makes redaction cost nothing.

Does this apply to geometry? #

Usually not — parcel boundaries are published cadastral data and carry no personal information. The exception is geometry that reveals something else, such as a precise structure footprint for a protected address, and in those cases the parcel should be excluded from shared artefacts rather than have its geometry blurred, since blurred geometry is both identifying and wrong.

How do I redact data already in historical logs? #

Log retention is the practical answer. Rewriting historical logs is difficult to do completely — copies exist in aggregation services, backups and tickets — so the workable approach is to apply redaction going forward, shorten retention on the unredacted archive, and treat the archive as sensitive for as long as it exists.

What key rotation strategy makes sense? #

Rotate infrequently and record the epoch with each log, because rotating the key changes every pseudonym and breaks correlation across the boundary. Annual rotation with the key epoch recorded in the run manifest keeps correlation useful within a year and limits the value of a compromised key.

Does redaction affect the audit trail’s completeness? #

No, provided the trail identifies parcels rather than people. A determination’s provenance needs the parcel identifier, the rule version, the input hashes and the measurements — none of which are personal. If an audit trail genuinely requires an owner’s identity, it belongs in the controlled store with the certificate, not in the log, which is the split capturing CRS provenance in validation logs already assumes.

Should the redaction filter be shared across projects? #

The mechanism yes, the policy no. The filter and the pseudonym function are generic; FIELD_POLICY is specific to a schema and to a jurisdiction’s rules, and a shared policy is one that is wrong somewhere. Keep the policy next to the schema it classifies, and treat a schema change without a policy change as an incomplete change.

Related #

Part of: Validation log design

• Structured JSON logging for geospatial pipelines — the log this filters.
• Capturing CRS provenance in validation logs — what a log should carry instead.
• Writing golden-file tests for zoning rule packs — fixtures that must be redacted before they are committed.
• Digitally signing compliance certificates in Python — the artefact that identifies a parcel by design.
• Tracking data lineage across geospatial ETL steps — provenance without personal data.