Data sources

Where every dataset comes from — the access path, the grain, and the join keys for each source system the models read.

Every 8020IQ model is built on a small set of source systems. This page is the map of each one: the system it lives in, how we reach it, the grain it arrives at, and the key it joins on. It is the provenance index behind how we match permits to properties and the gold variable inventory — and the Overview's answer to "where do the data come from?".

AWS data lake — three layers

The property + permit substrate lives in S3, read-only under the 8020rei-prod / 8020rei-coo profiles. Three layers, gold → silver → bronze, in decreasing degrees of curation.

Lake layout reference: exact partition paths — gold Roofing_Master_Gold_V1_*/<FIPS>, silver Use_Type and REM/<period>/<FIPS> — are the working strings used by the pipeline. The bronze scoped-read role name is not reproduced verbatim here; confirm against the lake-layout note before wiring a new reader. See engine transparency for how the layers feed the model.

BigQuery — feedback loop

The client-feedback loop (which properties actually transacted / were worked) is mirrored into BigQuery, separate from the S3 lake.

Known caveats (carry forward): the feedback zip column strips leading zeros (07002 → 7002), so any zip-join must LPAD both sides to 5 digits or every Northeast (0xxxx) row silently drops. The feedback event_date is also unreliable (future / 1972 / back-dated values) — do not use it as a clean time anchor without scrubbing.

Vendors

Two licensed vendors supply the raw property and permit signal that the lake curates.

BZ permits are building-level, so the address standardizer is what carries a permit to the right parcel — see how we match and permit classification for the type/subtype taxonomy.

US demographics cache

A free-government, nationwide parquet cache at data/demographics/, curated to property-buyer-signal variables and joinable to property records on standard zero-padded FIPS string keys. Pulled once, reused across every model and every model version — not specific to one build.

ZCTA ≠ USPS ZIP (~3–5% mismatch — bridge via the HUD ZIP↔ZCTA crosswalk). ACS uses -666666688 as a "not available" sentinel — filter it before use. For T0 below a source's safe cutoff, fall back to the prior vintage or NaN. The cache also holds many more free-government families beyond the eight above (BEA, CDC, EPA, FEMA, FHFA, NOAA, USDA, Redfin/Zillow time-series, etc.); the eight here are the load-bearing demographic tiers. This #demographics section is the deep-link target for the data manifest.

County-portal scraping

Direct-from-county source discovery to fill BuildZoom coverage gaps and add distress signal not present in licensed feeds. Stage 1 only — source discovery, no production scrape yet.

Stage 1 catalogs each (county × datatype) source, proves reachability and scores viability. Stage 2 (pilot scrape) is gated on counsel review of bulk-scrape ToS + a QA pass; nothing here ships into a model today. Court records default to skip pending legal review. Cross-validation against the BuildZoom gold master is the eventual reconciliation check.

Conventions across every source

Rendered from CLAUDE.md external-context + notes/demographics/README.md + notes/scraping/ · 2026-06-02