Ingesting ArcGIS Feature Services into a Compliance Pipeline

Most municipalities that publish zoning at all publish it as an ArcGIS feature service, which is the best source available — it is authoritative, current, and typed. It is also live, which is the problem: a compliance run against a URL is a run against whatever that URL returned at the moment it was called, and that is not something anyone can re-run. This guide reads a feature service completely and correctly — paginating past the record limit, preserving field types through the JSON round-trip, and writing an immutable snapshot the run is actually evaluated against. It is the live-source case of zoning layer ingestion strategies.

Prerequisites #

• The service’s layer URL, ending in a layer index (e.g. /FeatureServer/0), not the service root
• requests, GeoPandas 1.0+ and PyProj 3.4+
• Somewhere immutable to write the snapshot — object storage, or a content-addressed local path
• The service’s terms of use, and a polite request rate
• A projected CRS decided in advance, per choosing a projected CRS

Step-by-step #

Step 1: Read the service metadata before reading any features #

The layer’s own description tells you the record limit, the field types, the geometry type and the spatial reference. Every subsequent decision depends on it, and skipping it is how people end up with a truncated layer they believe is complete.

import requests

def layer_metadata(layer_url: str) -> dict:
    r = requests.get(layer_url, params={"f": "json"}, timeout=60)
    r.raise_for_status()
    meta = r.json()
    if "error" in meta:
        raise RuntimeError(meta["error"])
    return {
        "name": meta["name"],
        "max_record_count": meta.get("maxRecordCount", 1000),
        "supports_pagination": "Pagination" in meta.get("advancedQueryCapabilities", {}),
        "wkid": meta["extent"]["spatialReference"].get("latestWkid")
                or meta["extent"]["spatialReference"]["wkid"],
        "oid_field": meta.get("objectIdField", "OBJECTID"),
        "fields": {f["name"]: f["type"] for f in meta["fields"]},
    }

maxRecordCount is the number that matters most. It is commonly 1000 or 2000, it is enforced silently, and a query for a layer of 40,000 parcels returns the first 1000 with no error and no indication that anything is missing.

Step 2: Paginate by object ID, not by offset #

Services advertise resultOffset pagination, and it works — until the underlying data changes between pages, at which point rows shift and you silently skip or duplicate features. Paginating by object-ID range is stable against concurrent edits because each page is defined by a value range rather than by a position.

def object_ids(layer_url: str, where: str = "1=1") -> list:
    """Every object ID in one call — this endpoint is not subject to maxRecordCount."""
    r = requests.get(f"{layer_url}/query",
                     params={"where": where, "returnIdsOnly": "true", "f": "json"},
                     timeout=120)
    r.raise_for_status()
    return sorted(r.json()["objectIds"])

def id_batches(ids: list, size: int):
    for i in range(0, len(ids), size):
        yield ids[i:i + size]

The returnIdsOnly endpoint is exempt from the record limit on essentially every service, so it gives you the complete population up front — which also means you know how many features you should end up with, and can assert it.

Step 3: Fetch each batch, asking for the geometry you actually want #

Request the features in the spatial reference you intend to work in and let the service reproject; it is authoritative about its own data and it saves a client-side transform whose provenance you would otherwise have to record. Ask for GeoJSON where supported, and fall back to Esri JSON where it is not.

import geopandas as gpd

def fetch_batch(layer_url, ids, oid_field, out_wkid, fields="*"):
    params = {
        "where": f"{oid_field} >= {ids[0]} AND {oid_field} <= {ids[-1]}",
        "outFields": fields,
        "outSR": out_wkid,
        "returnGeometry": "true",
        "geometryPrecision": 6,     # decimals; keep well below survey accuracy
        "f": "geojson",
    }
    r = requests.get(f"{layer_url}/query", params=params, timeout=180)
    r.raise_for_status()
    payload = r.json()
    if "error" in payload:
        raise RuntimeError(payload["error"])
    return gpd.GeoDataFrame.from_features(payload["features"], crs=f"EPSG:{out_wkid}")

geometryPrecision is worth setting deliberately. Left unset, services return full double precision and the response can be several times larger than it needs to be; set too low, it quantises boundaries enough to open gaps between neighbouring parcels — the defect described in snapping slivers between adjacent parcels. Six decimal places in a projected frame is far below any survey accuracy and safe.

Step 4: Restore the field types the JSON round-trip destroyed #

GeoJSON has no date type and no integer/float distinction beyond what the parser infers, so an Esri date field arrives as an epoch-milliseconds integer and a code field of "01" may arrive as the number 1. Both cause failures much later, where they are hard to attribute.

import pandas as pd

ESRI_DATE = "esriFieldTypeDate"
ESRI_STRINGY = {"esriFieldTypeString", "esriFieldTypeGUID", "esriFieldTypeGlobalID"}

def restore_types(gdf, field_types: dict):
    """Put back what GeoJSON cannot carry: dates, and string codes that look numeric."""
    for name, esri_type in field_types.items():
        if name not in gdf.columns:
            continue
        if esri_type == ESRI_DATE:
            gdf[name] = pd.to_datetime(gdf[name], unit="ms", utc=True, errors="coerce")
        elif esri_type in ESRI_STRINGY:
            gdf[name] = gdf[name].astype("string")
    return gdf

The string case is the one that bites in zoning work specifically: district codes are frequently zero-padded, and a zoning_district of "01" silently becoming 1 breaks every applicability predicate that compares against the padded form — with no error, because a predicate that matches nothing simply applies to nothing.

Step 5: Write a snapshot and evaluate against that #

This is the step that converts a live read into a reproducible one. Write the assembled layer once, hash it, and record the hash in the run manifest; from that point the pipeline reads the snapshot, never the service.

import hashlib
import json
from datetime import datetime, timezone

def snapshot(gdf, meta: dict, layer_url: str, out_dir: str) -> dict:
    path = f"{out_dir}/{meta['name']}.parquet"
    gdf.to_parquet(path, index=False)
    digest = hashlib.sha256(open(path, "rb").read()).hexdigest()
    manifest = {
        "source_url": layer_url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "feature_count": len(gdf),
        "crs": gdf.crs.to_string(),
        "content_sha256": digest,
        "service_fields": meta["fields"],
    }
    with open(f"{out_dir}/{meta['name']}.manifest.json", "w") as fh:
        json.dump(manifest, fh, indent=2)
    return manifest

GeoParquet is a good snapshot format for the reasons set out in converting parcel shapefiles to GeoParquet for batch runs: it preserves types, it stores the CRS, and it reads back in a fraction of the time. The manifest is what lets tracking data lineage across geospatial ETL steps say which version of the district layer produced a given verdict.

Step 6: Detect when the service has changed under you #

Re-ingesting on a schedule is routine; noticing what changed is the useful part.

def compare_snapshots(old: dict, new: dict) -> dict:
    return {
        "content_changed": old["content_sha256"] != new["content_sha256"],
        "feature_delta": new["feature_count"] - old["feature_count"],
        "crs_changed": old["crs"] != new["crs"],
        "fields_added": sorted(set(new["service_fields"]) - set(old["service_fields"])),
        "fields_removed": sorted(set(old["service_fields"]) - set(new["service_fields"])),
    }

A removed field is the one to alarm on. Agencies rename columns without notice, and a rule pack whose predicates reference the old name will stop matching anything the moment the rename lands.

Verification #

The ingest is correct when the feature count matches the service’s own count, no geometry is null, and the CRS is the one that was asked for.

expected = requests.get(f"{layer_url}/query",
                        params={"where": "1=1", "returnCountOnly": "true", "f": "json"},
                        timeout=60).json()["count"]

assert len(gdf) == expected, f"ingested {len(gdf)} of {expected} features"
assert gdf.geometry.notna().all(), "null geometry in the ingested layer"
assert gdf.geometry.is_valid.all() or repaired, "invalid geometry needs a repair pass"
assert gdf.crs.to_epsg() == out_wkid

The count assertion is the whole point of the exercise. It is the check that catches a silent truncation at the record limit, which is the single most common way an ArcGIS ingest goes wrong, and it costs one extra request.

Common Pitfalls #

• Assuming one query returned everything. maxRecordCount truncates without an error. Always compare against returnCountOnly.
• Offset pagination against a live service. Rows shift when the source is edited mid-read, producing duplicates and gaps that are nearly impossible to spot afterwards.
• Letting GeoJSON flatten a zero-padded code to an integer. The applicability predicates then match nothing, and a rule that matches nothing looks like a clean run.
• Evaluating against the URL rather than the snapshot. A verdict that cannot be reproduced is not evidence, and the service will have changed by the time anybody asks.
• Ignoring exceededTransferLimit. Some services return this flag in the response rather than an error; if it is present, the page is incomplete regardless of what the row count suggests.

Frequently Asked Questions #

Should I use the ArcGIS Python API instead of raw REST? #

If it is already in the environment, yes — it handles pagination and typing for you. The reason to know the REST calls anyway is that the failure modes above are properties of the service, not of the client, and a library that hides pagination also hides a truncation when something goes wrong. Whichever client is used, the count assertion stays.

What about services that require authentication? #

Token-based auth adds a token parameter to every request and an expiry to manage; the pagination and typing logic is unchanged. Keep the token out of the manifest and out of logs — it is exactly the kind of value that redacting sensitive parcel data from logs exists to keep out of a shared artefact.

How often should the snapshot be refreshed? #

As often as the source changes and no more. Zoning district layers change a few times a year; parcel fabrics are usually refreshed quarterly or monthly. Ingesting nightly against a layer that changes annually generates 364 identical snapshots and one that matters, and the content hash makes it cheap to keep only the ones that differ.

Can the service do the spatial filtering for me? #

Yes — a geometry plus spatialRel parameter will restrict the query to an envelope, which is worth doing when the pipeline only covers part of a county. It does not remove the need for pagination, and it introduces a subtlety: the filter is applied in the service’s spatial reference unless you say otherwise, so an envelope in the wrong frame quietly returns the wrong parcels.

What if the service is slow or unreliable? #

Batch by object ID as above, retry each batch with backoff, and keep the batches small enough that a retry is cheap — a few hundred features rather than the full record limit. Because the batches are defined by ID range rather than position, a retried batch is exactly the same batch, so retries are safe in a way that offset pagination’s are not.

Is the service’s own reprojection trustworthy? #

Generally yes, and it is preferable to a client-side transform because the service knows its own datum and any published transformation grid. The exception is a datum change — asking a NAD83 service for WGS84 output invokes a transformation whose method you did not choose, which is precisely the situation handling datum shifts NAD83 to WGS84 warns about. Request a projected frame on the same datum and the reprojection is safe.

Should the snapshot keep every field? #

Keep everything the service publishes, even fields the pipeline does not read. Storage is trivially cheap relative to the cost of re-ingesting a historical state that no longer exists, and the field somebody needs in eighteen months is reliably the one that was dropped as unnecessary today.

Related #

Part of: Zoning layer ingestion strategies

• How to parse municipal zoning PDFs into GeoJSON — the source to fall back on when no service exists.
• Converting parcel shapefiles to GeoParquet for batch runs — the snapshot format.
• Validating CRS metadata before a compliance run — checking what the service actually returned.
• Fixing invalid parcel polygons with make_valid — the repair pass that follows ingest.
• Tracking data lineage across geospatial ETL steps — where the manifest goes.