Cloud-Native Geospatial Formats for Compliance Pipelines

Most compliance pipelines spend more time reading data than measuring it, and the reason is usually the format. A shapefile has to be walked from the beginning; a GeoParquet file with row-group statistics can be read for one tile without touching the rest. This module of Spatial Analysis Pipelines for Density & Proximity Checks covers what the cloud-native formats actually change, which ones belong where in a compliance stack, and how to adopt them without losing the reproducibility the audit trail depends on.

Prerequisites #

• A working pipeline whose read stage has been measured, not assumed. Adopting a format to fix a bottleneck that turns out to be elsewhere is a common and expensive detour.
• pyogrio or pyarrow alongside GeoPandas 1.0+, and rasterio 1.3+ for the raster side.
• Object storage, or a filesystem that supports range reads. The formats’ main advantage is reading part of a file, which a transport that cannot seek discards.
• The snapshot discipline from zoning layer ingestion strategies: whatever the format, the run reads a hashed copy rather than a live source.
• A decision about where converted copies live, since a derived format is a second artefact that needs its own identity.

What Actually Changes #

The phrase “cloud-native” covers several distinct properties, and only some of them matter for compliance work.

Partial reads. The important one. A format with an internal index and range-readable layout lets a worker fetch the rows or pixels covering its tile and nothing else. On a partitioned county run this is the difference between every worker reading the whole county and each reading its own share.

Columnar storage. GeoParquet stores each attribute contiguously, so reading three columns of a forty-column parcel table costs three columns of I/O. Compliance runs almost always want a handful of attributes and the geometry, which makes this a larger saving than it sounds.

Compression that survives seeking. Per-row-group or per-tile compression keeps files small without forcing a full decompress to reach the middle.

Self-description. CRS, schema and statistics travel inside the file. That removes the missing-.prj failure mode entirely and makes a snapshot genuinely self-contained — which matters more for an archived compliance artefact than for a working file.

What does not change is the geometry engine. The predicates still run in GEOS at the same speed; these formats make getting the data to GEOS cheaper.

Choosing a Format Per Role #

Compliance stacks have three distinct storage roles and the right format differs for each.

For the working vector store — parcels, districts, overlays that every run reads — GeoParquet is the strong default. It is columnar, partitionable, compresses well, carries its CRS, and reads natively into GeoPandas. Partition it by the same spatial tiles the run uses and each worker’s read becomes a directory listing plus one file.

For raster inputs — impervious surface, canopy, lidar-derived surfaces — a cloud-optimized GeoTIFF is the equivalent: internally tiled, with overviews, so a window read fetches only the tiles it overlaps. The overviews are worth building even when the analysis uses full resolution, because every review map and thumbnail then costs a fraction of the full read.

For interchange and web delivery — handing results to another agency, or feeding a review map — FlatGeobuf is a good fit: streamable, indexed, and readable incrementally, so a map can start drawing before the file has finished arriving. It is a poor working store, being row-oriented, and an excellent delivery format.

The one thing to avoid is treating a delivery format as a working store or vice versa. A compliance run against a FlatGeobuf pays row-oriented costs on every attribute read; a review map served from GeoParquet has to fetch far more than it draws.

Conversion Without Losing the Original #

A converted file is a derived artefact, and derived artefacts have a way of becoming the only copy.

The discipline that keeps this safe is short. Keep the supplier’s original bytes, hashed, in the snapshot store — that is what the audit trail refers to and what a dispute is resolved against. Record the conversion as a lineage step with its parameters. And derive the working copy from the snapshot rather than from a live source, so the two cannot diverge.

def snapshot_and_convert(src_path, store, tile_col="tile_id"):
    """Original preserved and hashed; working copy derived from it, not from the source."""
    digest = sha256_file(src_path)
    store.put(f"snapshots/{digest}", src_path)          # supplier's bytes, untouched

    gdf = gpd.read_file(f"snapshots/{digest}", engine="pyogrio")
    gdf = gdf.to_crs(WORKING_CRS)
    gdf[tile_col] = assign_tiles(gdf.geometry)          # partition on the run's own tiles
    gdf.to_parquet(f"working/{digest}", partition_cols=[tile_col],
                   compression="zstd", write_covering_bbox=True)
    return {"source_sha256": digest, "working": f"working/{digest}",
            "crs": WORKING_CRS, "partitioned_by": tile_col}

Keying the working copy on the source digest is the detail that makes this reproducible: two runs referencing the same snapshot necessarily read the same working copy, and a re-conversion with different parameters lands somewhere else rather than overwriting.

Raster Inputs and the Windowed Read #

Vector data gets most of the attention, but the raster side is where the format choice makes the largest single difference, because raster files are large and compliance rarely needs all of one.

An impervious-surface layer for a county is gigabytes; the question asked of it is almost always “what is the impervious fraction inside this parcel”, which touches a few hundred pixels. A plain GeoTIFF answers that by reading enough of the file to reach those pixels; a cloud-optimized GeoTIFF, being internally tiled, reads the tiles that overlap the parcel and nothing else. On a run evaluating ten thousand parcels the difference is not incremental.

import rasterio
from rasterio.mask import mask

def impervious_fraction(cog_path, parcel_geom):
    """Windowed read: only the internal tiles overlapping this parcel are fetched."""
    with rasterio.open(cog_path) as src:
        data, _ = mask(src, [parcel_geom], crop=True, filled=False)
        valid = data.compressed()
        if valid.size == 0:
            return None                       # no coverage — a finding, not a zero
        return float((valid > 0).sum() / valid.size)

Two details matter for compliance rather than performance. The None return on no coverage is deliberate: a parcel outside the raster’s extent has an unknown impervious fraction, and returning zero would silently report it as fully pervious. And the raster’s own resolution bounds the precision of the answer — a thirty-metre grid cannot resolve a small residential lot’s coverage meaningfully, and a verdict computed from it should carry that limitation rather than a percentage to two decimal places.

Overviews are worth building even when the analysis reads full resolution, because every map, thumbnail and review image then costs a fraction of a full read. They also make a quick visual sanity check cheap, which is the step most often skipped when it is slow.

What Adoption Is Actually Worth #

The honest answer is that it depends on where the time currently goes, which is why the measurement comes first.

On a pipeline reading shapefiles from a network mount, converting the working store to partitioned GeoParquet routinely removes the majority of wall time, because the read was the majority of wall time. On a pipeline already reading from a local GeoPackage with a spatial index, the gain is real but modest. And on a pipeline whose cost is dominated by geometric predicates over pathologically complex parcels, the format changes nothing at all.

The secondary benefits are more uniform. Self-describing files remove a class of ingestion failure. Columnar storage makes ad-hoc analysis over the working store pleasant rather than punitive. And partitioned layout makes the spatial chunking described in batch processing optimization natural rather than something bolted on.

Set against that, the costs are modest but real: another format in the stack, a conversion step to operate, tooling that some colleagues will not have, and a specification that is younger than shapefile by three decades and still occasionally surprising at the edges.

Partitioning the Working Store #

A cloud-native format only pays if the layout matches how the data is read, and for compliance work the access pattern is almost always spatial.

The natural partition is the same tile scheme the run uses, which makes the read for a tile a directory selection rather than a scan. Partition sizes want to land in the tens of megabytes: much smaller and the per-file overhead dominates, much larger and a worker reads far more than its tile. For a county parcel fabric that usually means a few hundred partitions, which is comfortable for both object storage and a filesystem.

Adding a bounding-box column alongside the geometry — supported directly by recent GeoParquet writers — lets a reader prune row groups within a partition too, so even a coarse partitioning gets fine-grained skipping. It costs four floats per row and is worth it on any layer large enough to be worth converting.

def read_tile(working_path, tile_id, columns, bbox=None):
    """Read one tile's parcels and only the columns the rules need."""
    return gpd.read_parquet(
        working_path,
        filters=[("tile_id", "==", tile_id)],   # partition pruning
        columns=[*columns, "geometry"],          # column pruning
        bbox=bbox,                               # row-group pruning within the tile
    )

Two habits keep this honest. Verify pruning by measuring bytes read rather than wall time, because a warm cache makes an unpruned read look fast. And re-partition when the tile scheme changes rather than leaving the working store partitioned on a scheme nobody uses any more, which quietly reverts every read to a scan.

Attribute layout deserves one thought as well: put the columns rules actually read — district code, unit counts, use class — in the working copy, and leave the forty descriptive fields nobody evaluates in the archived original. A narrower working store is faster to read and easier to reason about, and the full record is a snapshot away if anybody needs it.

Migrating an Existing Pipeline #

Adoption goes badly when it is attempted as a rewrite and well when it is attempted as a substitution behind an unchanged interface.

The interface worth preserving is the read function: whatever the pipeline calls to get a tile of parcels. If that call already takes a tile identifier and returns a GeoDataFrame, the format change is entirely inside it, and every stage downstream is untouched. If it does not — if stages read files directly — introducing that function is the first step and is worth doing on its own merits regardless of format.

With the seam in place, migrate one layer at a time and compare. Run the pipeline against the old and new working stores over the same corpus and diff the verdicts; they must be identical, because a format change is not supposed to change anything. A difference means the conversion altered geometry — usually through an unintended reprojection or a precision change — and finding that on one layer is much better than finding it after converting six.

Keep the old path available until the new one has run in production for a cycle or two. Reverting a format migration should be a configuration change rather than a restore, and it will not be if the original working store was deleted the day the new one worked.

Budget for the boring parts: a conversion job that runs on refresh, monitoring that it completed, and a check that the converted copy’s feature count matches the source. Those three account for most of the operational cost, and skipping them is how a stale working copy ends up silently serving a compliance run.

Compatibility and the Long Archive #

A compliance artefact may need to be readable in a decade, which is an unusual requirement for a data format and worth thinking about explicitly.

The safe arrangement is to keep the archived snapshot in the format the supplier published — whatever that was — and to treat the cloud-native copy as a working derivative that can be regenerated. If GeoParquet’s specification evolves in a way that breaks an old file, the original is untouched and the working copy is rebuilt. Archiving only the derivative puts the long-term readability of your evidence on the youngest component in the stack.

Where a converted file is archived, record the format version alongside the data, and prefer widely-implemented options over exotic ones — standard compression codecs, no unusual encodings — since the constraint a decade out is what other software can read, not what yours could write.

A last practical note on tooling. Colleagues who open these files in desktop GIS will have a mixed experience: recent QGIS reads GeoParquet and cloud-optimized GeoTIFF comfortably, older installations and some proprietary tools do not. That is an argument for keeping a conventional export available for interactive use rather than for avoiding the formats in the pipeline — the working store is read by code, and the copy a colleague opens can be generated on request from the same snapshot.

Troubleshooting #

• Reads are no faster after conversion. The transport does not support range requests, or the file is one row group. Check the row-group count and the storage layer before blaming the format.
• GeoParquet loses the CRS. An older writer omitted the geo metadata. Assert the CRS on read and fail rather than defaulting; a silently missing frame is the failure this format was meant to eliminate.
• COG window reads pull the whole file. Missing internal tiling or overviews. A GeoTIFF is only cloud-optimized if it was written that way; validate before trusting it.
• Partition pruning does not happen. Filters applied after loading rather than pushed into the read. Pass the filter to the reader, and verify by watching bytes read rather than wall time.
• Two runs read different working copies. The working copy was keyed on a name rather than on the source digest, and a re-conversion overwrote it.

The same reasoning applies to any downstream consumer with fixed tooling: serve them the format they can read, and keep the pipeline on the format it reads fastest.

Treated as a substitution behind a stable read interface rather than as a re-platforming, the whole adoption is a contained piece of work with a measurable result and an easy retreat — which is the profile of a change genuinely worth making, as distinct from one that is merely worth debating at length.

Related #

Part of: Spatial analysis pipelines for density and proximity checks

• Converting parcel shapefiles to GeoParquet for batch runs — the conversion, step by step.
• Reading cloud-optimized GeoTIFFs for impervious surface checks — the raster side.
• Batch processing optimization — where the read cost sits in a whole run.
• Zoning layer ingestion strategies — snapshotting the original before converting it.
• Provenance and lineage tracking — recording the conversion as a lineage step.

Conclusion #

Cloud-native formats do not make geometry faster; they make getting to the geometry cheap, which on most compliance pipelines is where the time actually goes. Measure first, then convert the working store to partitioned GeoParquet, use cloud-optimized GeoTIFFs for raster inputs and FlatGeobuf for delivery. Keep the supplier’s original bytes hashed and archived, derive the working copy from that snapshot, and key it on the source digest. The run gets faster, the ingestion failures disappear, and the evidence still rests on the file the agency actually published.