Rule Storage Formats: JSON vs YAML vs Database

Where you keep zoning rules quietly determines how easily planners edit them, how safely engineers version them, and how fast the engine evaluates them across a jurisdiction. This module of Rule Engine Design for Zoning & Setback Automation compares three durable options for persisting setback, height, and overlay constraints: flat JSON files, human-friendly YAML files, and a relational or PostGIS-backed database. The goal is a storage layer that keeps rule definitions decoupled from execution logic while matching your team’s editing habits, review process, and dataset scale.

Prerequisites #

Before committing to a storage layer, make sure the surrounding pipeline can support any of the three options:

• A normalized rule schema with stable field names such as rule_id, zone_class, setback_m, precedence, and effective_date.
• A validation library in place: jsonschema for JSON, pydantic or cerberus for YAML-derived dicts, or database constraints for the relational path.
• Python 3.10+ with pyyaml (use yaml.safe_load), plus sqlalchemy and a PostGIS-enabled PostgreSQL instance if you intend to test the database route.
• A version-control workflow (Git) for file-based formats, or migration tooling such as Alembic for the database.
• A rule count estimate and expected edit frequency, because scale and concurrency are the two criteria that most often decide the outcome.

Comparison Criteria #

The three formats are not ranked absolutely; each optimizes a different constraint. The table below scores them across the six criteria that matter most when rules must survive council amendments, staff turnover, and growing parcel volumes.

The pattern is consistent: file formats win on transparency and review ergonomics, while the database wins on query power, concurrency, and scale. YAML edges out JSON for anything a non-developer will touch because comments let you annotate the statutory citation next to each threshold.

Core Workflow #

Whichever format you choose, the engine should never evaluate rules straight off disk or the wire. Load once, validate, and convert to a single in-memory intermediate representation so downstream code stays storage-agnostic. The following loader normalizes YAML and JSON into identical dictionaries and enforces a schema before any rule reaches the evaluator.

import json
from pathlib import Path
import yaml  # PyYAML
from pydantic import BaseModel, Field, ValidationError

class ZoningRule(BaseModel):
    rule_id: str
    zone_class: str
    setback_m: float = Field(ge=0)   # linear metres, projected CRS assumed
    precedence: int = 0
    effective_date: str

def load_rules(path: str) -> list[ZoningRule]:
    text = Path(path).read_text(encoding="utf-8")
    # safe_load blocks arbitrary object construction; json.loads for .json
    raw = yaml.safe_load(text) if path.endswith((".yaml", ".yml")) else json.loads(text)
    rules = []
    for entry in raw["rules"]:
        try:
            rules.append(ZoningRule(**entry))   # validation happens here
        except ValidationError as exc:
            raise ValueError(f"Invalid rule {entry.get('rule_id')}: {exc}") from exc
    return rules

Because both branches emit the same ZoningRule objects, you can migrate from files to a database later without rewriting the evaluator. The database path simply swaps the file read for a query and passes each row through the same model.

Implementation Patterns #

For file-based storage, keep rules in one document per jurisdiction rather than one giant file, so diffs stay scoped and merge conflicts stay rare. Store distances in metric units to match the projected CRS your spatial engine uses, and never rely on geographic degrees for setback thresholds. The database pattern below reads active rules for a zone and joins them directly to parcel geometry, which is where relational storage decisively outperforms flat files.

import geopandas as gpd
from sqlalchemy import create_engine

engine = create_engine("postgresql://gis:secret@localhost:5432/compliance")

def load_active_rules(zone_class: str, as_of: str) -> gpd.GeoDataFrame:
    # Indexed filter on zone_class and effective_date runs in the database,
    # so only relevant rows cross the wire even for millions of rules.
    sql = """
        SELECT rule_id, zone_class, setback_m, precedence, geom
        FROM zoning_rules
        WHERE zone_class = %(zone)s AND effective_date <= %(as_of)s
        ORDER BY precedence DESC
    """
    gdf = gpd.read_postgis(sql, engine, geom_col="geom",
                           params={"zone": zone_class, "as_of": as_of})
    return gdf.to_crs("EPSG:26910")  # project to a metric CRS before any distance op

Relational storage also enforces referential integrity: a foreign key from zoning_rules.overlay_id to an overlay table prevents dangling references a hand-edited YAML file would accept. The transactional guarantees pair naturally with Async Rule Execution Patterns, where workers read a consistent rule snapshot while edits land safely in another transaction.

Edge Cases & Data Repair #

Storage format changes the failure modes you guard against. YAML silently coerces unquoted values, so a zone code like NO parses as boolean False and a bare 1.10 loses its trailing zero — quote categorical fields and validate types on load. JSON has no comment channel, so provenance notes must live in a dedicated _meta field. Databases invert the risk: a rule can be deleted without an obvious diff, so add an append-only history table to preserve prior versions.

def normalize_types(entry: dict) -> dict:
    # Guard against YAML's implicit typing of codes and numbers.
    entry["zone_class"] = str(entry["zone_class"])          # 'NO' must stay a string
    entry["setback_m"] = round(float(entry["setback_m"]), 3)  # stable numeric precision
    if entry.get("precedence") is None:
        entry["precedence"] = 0                             # explicit default beats null
    return entry

For irregular parcel geometries feeding the buffer stage, the same discipline that governs Dynamic Setback Buffer Generation applies here: repair geometry before evaluation and never let a malformed rule row reach the buffer routine.

Audit Logging & Provenance #

Regardless of format, every evaluation must be reproducible from recorded provenance. Log the storage source or database snapshot identifier, the rule version or commit hash, the projected CRS in use, and the schema version the loader validated against. File formats make this nearly free: pin the Git commit that supplied the rules. Databases need explicit effort — capture the transaction timestamp and the effective_date filter so a later audit can reconstruct exactly which rows were active. Store these fields alongside the compliance result, not in a separate log that can drift out of sync.

Troubleshooting #

• Distorted setbacks after a format switch: the new source stored distances in feet while the engine assumed metres. Standardize units in the schema and validate ranges on load.
• YAML edit silently ignored: indentation placed the key at the wrong nesting level. Run yaml.safe_load in a pre-commit check and assert the expected top-level keys exist.
• Database rules stale in cache: workers cached a rule snapshot and missed an amendment. Version the snapshot and invalidate on effective_date changes rather than caching indefinitely.
• Noisy JSON diffs on review: key reordering by a formatter obscures the real change. Sort keys deterministically on write so diffs show only substantive edits.

Choosing by Change Rate and Review Path #

The format debate usually gets conducted on technical grounds — parsing speed, tooling, whether comments are supported — and then loses to the two questions that actually decide it: how often do these rules change, and who has to approve the change?

Rules that change a few times a year and require a planner’s sign-off want a text file in version control, because the review artefact is a diff and the approval record is a merge. Rules that change many times a day and are approved by nobody — a per-tenant tolerance that operations staff tune, say — want a database row, because a pull request per change is friction with no reviewer to justify it. Most systems have both kinds and are best served by both stores, provided each rule has exactly one home and the pipeline knows which.

The mistake worth naming is treating the database as an authoring surface for regulation. A rules table that anybody with write access can update has no review, no diff, no history beyond an updated-at column, and no answer to “who changed this and on whose authority” — and those are precisely the questions asked when a verdict is challenged. If regulation must live in a database for query reasons, load it there from version-controlled text and make the table read-only to the application, so the text stays the source of truth and the table is a projection of it.

def load_rule_packs(text_dir, db, git_sha: str):
    """Text files are the source of truth; the table is a rebuilt projection.

    Loading is idempotent and stamped with the commit, so any row in the table
    can be traced back to a reviewed change.
    """
    packs = [parse(p) for p in sorted(text_dir.glob("*.yaml"))]
    with db.transaction():
        db.execute("DELETE FROM rule_packs WHERE source = 'repo'")
        db.executemany(INSERT_PACK, [{**p, "source": "repo", "git_sha": git_sha}
                                     for p in packs])
    return len(packs)

Serialisation format then matters far less than it seems. YAML earns its place for hand-authored rules because it takes comments — and the comment explaining why a threshold is unusual is often the most valuable line in the file. JSON is the better wire and storage format precisely because it has no comments and no ambiguity: no accidental octal, no Norway problem, no significant whitespace. Authoring in YAML and storing the canonical JSON, generated at load time, gets both properties without a debate.

Schema Validation as the Gate #

Whatever the store, the load path needs one gate that a malformed rule cannot pass, and it should reject the pack rather than skipping the rule. Skipping a single bad rule leaves a pipeline running with a silently incomplete rule set, which produces verdicts that look complete and are not — the worst available outcome.

A useful schema goes beyond types. Requiring a unit on every measured value catches the most expensive class of error at load time rather than in a report. Requiring a citation makes an unattributable rule impossible to ship. Constraining the operator to a small enumeration removes the ambiguity at the boundary. Requiring effective dates, and checking that no two versions of the same rule id have overlapping date ranges, catches the amendment that was edited in place instead of being closed and reopened.

Cross-rule checks are worth as much as per-rule ones and are cheap to add: every rule’s named measurement must exist in the measurement catalogue; every referenced district must exist in the controlled list; no rule id may be defined twice within a pack. Each of these is a one-line assertion over the loaded set and each corresponds to a real production failure — a renamed measurement, a district code retired by the agency, a copy-pasted block that overwrote its neighbour.

Run the same validation in three places: as a pre-commit hook, so the author finds the problem first; in continuous integration, so it cannot be bypassed; and at application startup, so a hand-edited file on a server is caught before it can produce a verdict. The check is identical each time, which means it can live in one function and be trusted equally in all three.

Migration between stores is easier than the debate suggests, provided the in-memory representation stays the same. Because the evaluator consumes validated rule objects rather than files or rows, swapping the loader is a contained change and can be done incrementally — a jurisdiction at a time, with both loaders running against the same corpus and their outputs compared. That comparison is worth doing even when no migration is planned: it is the cheapest available proof that the store is not quietly changing the rules it holds.

One further consideration decides more architectures than it should: how the rules are read at evaluation time. Rules are small — a jurisdiction’s complete rule set is measured in kilobytes — and they are read constantly, so they belong in memory for the duration of a run, loaded once and validated once. A pipeline that queries a rules table per parcel spends more time on rule lookup than on geometry, and, worse, opens the door to the rule set changing mid-run, which quietly destroys the run’s internal consistency.

Pin the version for the whole run instead. Load at start, record the version in the manifest, and evaluate every parcel against that snapshot even if an amendment lands in the store while the run is in flight. Half a county evaluated under one rule set and half under another is not a result anyone can defend, and the difference is invisible in the output unless the version was recorded per verdict — which is the argument for doing that too.

None of these choices is irreversible, which is worth remembering when the debate stalls.

Access control deserves a mention too. Whichever store holds the rules, the pipeline’s runtime credentials should be read-only against it. A compliance engine has no legitimate reason to write a rule, and removing the capability removes an entire category of incident — including the accidental one where a migration script run against the wrong environment rewrites a jurisdiction’s thresholds.

File layout is the last small decision with outsized effect. One file per district, named for the district, keeps diffs local and merge conflicts rare; one large file for everything guarantees that two people amending unrelated districts will collide. Keep shared definitions — measurement names, unit vocabulary, district lists — in their own file, since those change rarely and are referenced by all of the others.

Related #

Part of: Rule engine design for zoning and setback automation

• Migrating zoning rules from spreadsheets to YAML — getting existing rules into the store.
• Automating zoning code version control with Git — the review path that text storage buys.
• Spatial threshold configuration — the fields each stored record has to carry.
• Versioning rule references in audit trails — connecting a stored version to a verdict.

Recommendation #

Choose by scenario rather than preference. If planners edit rules directly and the set stays in the low thousands, use YAML files in Git — readable diffs and inline citations make council amendments reviewable, and the migration guide for moving zoning rules from spreadsheets to YAML shows how to get there cleanly. If rules are consumed primarily by machines and you want the strongest off-the-shelf validation, use JSON with JSON Schema. If you must join rules to hundreds of thousands of parcels, support concurrent editors, or query by spatial predicate, use a relational or PostGIS database with an append-only history. Many mature teams run a hybrid: author in YAML for reviewability, then compile validated rules into the database as the runtime source of truth.

Conclusion #

Rule storage is an architectural decision, not a file-format detail. YAML optimizes for the humans who amend ordinances, JSON optimizes for schema-driven interchange, and a database optimizes for query power, concurrency, and scale. By loading every format through one validating loader into a shared intermediate representation, you keep the evaluation core stable while retaining the freedom to migrate as your jurisdiction and parcel counts grow.