Define the universe of valid roof permits

For every Gold permit row, the Step 1.2 classifier emits one MECE column — permit_scope, a list of {type, action} items covering every (object, verb) the permit touches: 20 permit_type values × 7 permit_action values (see 1.2). The old flat 14-category enum (ROOFING_REPLACEMENT, … with a separate ROOFING sub-class) was superseded by this two-axis model in the v5 restructure — see the archived 14-cat spec. Output: data/sandbox/classify_v5/<FIPS>/labels.parquet; contract in 1.4.

Combine vendor coverage with reality · per-muni training-inclusion rule

For every (FIPS, jurisdiction, fa_muni), produce coverage_decision ∈ {INCLUDE, EXCLUDE, FLAG}. Only INCLUDED tuples feed training and delivery. False negatives are the dangerous failure mode → when in doubt, EXCLUDE. Output: evidence/sources/coverage/coverage_decisions.parquet.

As-of enrichment — property state at T0 DONE (3 counties) 2026-05-21

The training unit is the pair (property_id, T0) — not "a property with an event". Features anchor on T0 (the prediction moment) plus trend anchors T0−3 / T0−6; every feature value must be knowable at T0. The permit event never sets the feature anchor — it is only read forward from T0 to set the label (Step 4): y = 1 iff a qualifying roof event in (T0, T0+6]. As-of join takes the latest FA / Silver REM snapshot ≤ each anchor.

Leakage rule. "Past data" is not automatically safe. A feature is leaky if dated after that row's T0, even if still before today. The earlier "T-1 too close" framing was wrong: a refi one month pre-permit is a real, serve-time-available signal, not leakage — true leakage is a post-T0 feature or an FA snapshot vintage backfill. No artificial freshest-month exclusion.

Repair vs replacement. Different drivers — handled distinctly. Positive label = roof replacement (permit_action ∈ {REPLACEMENT, AMBIGUOUS}); REPAIR-only permits do not count as positives (they dilute the target). Prior repair history (n_roof_REPAIR_24m) is carried as a feature — a roof patched twice is near end of life.

Hurricane signal. Primary exogenous driver — storm → insurance claim → reroof, lagged months. Carry months_since_hurricane / in_hurricane_corridor; the lift is in the roof_age × months_since_hurricane interaction (seeded in Step 5).

Feature source. Internal feature families — property physical (incl. year built, roof age), distress, equity, owner state, sale recency, tenure — are reused from the Apollo macro-model feature builder. Shared dependency is the feature pipeline, NOT the Alpha model (roofing does not compete against Alpha). Reused features re-anchor to the roofing T0 and inherit the CLAUDE.md §7 leakage audit.

Decisions to lock: anchor set ({T0, T0−3, T0−6} · add T0−12?) · FA snapshot vintage audit (point-in-time integrity — gates whether near-T0 features need a guard) · negatives matched on month + FIPS, K-ratio (Step 4 cross-link).

Walk-forward folds & negative sampling PARAMS LOCKED 2026-05-21

Locked: horizon H = 6 months (label window (T0, T0+6]) · embargo 6 months = H (a shorter embargo leaves adjacent folds' label windows overlapping) · fold cadence 6 months (eval windows disjoint → honest variance estimate).

Fold layout — backward from the latest observation. latest_obs = most recent month with complete data. Fold x eval anchor E_x = latest_obs − 6·x; eval window (E_x, E_x+6]; fold x trains on all rows with T0_train ≤ E_x − 6. The first validatable fold is x=1 (latest_obs − 6) — latest_obs itself is the production scoring point, not a fold. Run as many folds as history allows (~6-8); fold x=1 is the most production-like. e.g. latest_obs = May 2026 → fold 1 eval (Nov 2025, May 2026], fold 2 (May 2025, Nov 2025], …

Negatives. Case-control: per positive, K negatives at the same FIPS + T0, from the coverage-INCLUDE SFH universe. Positive label = roof replacement (permit_action ∈ {REPLACEMENT, AMBIGUOUS}); a property supplies positive and negative rows at different T0s (discrete-time panel). K pending base-rate measurement.

Held-out final test stays carved out. The roofing model is judged on a walk-forward backtest (Step 9): lift over a random list of equal size. Baseline = the random list; roofing does not compete against Alpha (the "March 2025 vs Alpha" gate in CLAUDE.md is the macro sales model). Dev folds must not touch the backtest window's cells ± embargo. Open: pick a window old enough that its permit data is fully settled.

Decisions to lock: K negatives per positive (measure base rate first) · backtest window — old enough for settled permit data · positives-per-(FIPS, fold) count feeding the Step 6 per-FIPS-vs-global call.

Synthetic features · Internal × External SPEC · audited 2026-05-21

Feature columns on the enriched (property_id, T0) anchors from Step 3. Internal = property + owner (property physical incl. year built / roof age, distress, equity, tenure, sale recency, prior-permit history) — reused from the Apollo feature builder, not the Alpha model. External = market + environment.

Feature selection — expert pre-filter, then let the model judge. ~320 candidate columns need disciplined selection. Three steps: (1) expert pre-filter — drop obvious non-signal + no-variability columns, high confidence only; keep the "plausible but I doubt it" bucket. (2) quick model + permutation importance — multivariate, captures interactions; do NOT filter on univariate correlation, which misses roof_age × hurricane-type effects; drop importance ≈ 0. (3) train on the survivors. The distress family (~276 columns) is the main pre-filter target — collapse to a count + one trend + top-N flags. Drop the 2nd-diff for the first model.

External features split by whether they vary within (FIPS, T0). Step 4 matches negatives in the same FIPS + T0, so any feature constant across that cell — county macro (FRED rates, HPI, unemployment), a county-level hurricane flag — is identical for the positive and its negatives; the case-control sampling removes its signal entirely. Those do NOT go in the per-FIPS model — they belong at calibration / threshold tuning (Step 7 / 9). Useful External = features that vary within the cell: sub-county neighbour permit density, parcel-level storm exposure.

Hurricane must be parcel-level — peak wind / distance to track at the parcel, not a county-month flag (a county flag is wasted under T0-matched case-control). Neighbour permit density is a leakage trap — the rolling window must end at T0 and exclude the subject property.

Interactions — let the model do it. Gradient-boosted trees learn interactions automatically; hand-coding a boolean AND is redundant. Skip hand-coded interactions for the first model; reserve hand-coding for ratios.

Decisions to lock: feature budget number (after Step 4 reports positives-per-FIPS) · distress collapse set · insurance / claim climate source — sub-county grain, highest-value gap · FA ROOF_COVER availability for roof material.

Model training SPEC · audited 2026-05-21

One global model, not per-FIPS by default. Roof replacement is a rare event — a per-FIPS model on a small county (~100-300 positives/fold) re-learns the basics from thin data, while a global model learns them once from millions of rows. Default = one global gradient-boosted model with FIPS + region as features (the tree carves FIPS-specific behaviour where data supports it, pools where it does not). A dedicated per-FIPS model is a carve-out, justified only on evidence — not tied to the 5 gold-tier dev FIPS.

Algorithm. LightGBM (native categorical for FIPS) — confirm against HistGB / logistic on a roofing-specific arch sweep. The prior arch matrix was the sales model, not roofing.

Metrics — on the true base rate, not the case-control eval. The Step 4 eval is 1:K downsampled; AUC-PR read off it is inflated and top-decile recall is optimistic. Primary metrics must be computed on a holdout with the true population base rate. Success = lift over a random list of equal size on a walk-forward backtest (Step 9) — roofing does not compete against Alpha, the REI sales heuristic.

No re-weighting. The training table arrives already downsampled 1:K from Step 4 — Step 6 does not re-sample or re-weight; prior correction is Step 7's job. Tune hyperparameters once on pooled data and reuse — not a fresh 50-trial search per FIPS (overfits thin data). Early-stopping validation is carved from train, never the eval window.

Decisions to lock: global+FIPS-feature vs per-FIPS carve-out (+ row-count threshold) · algorithm (after a roofing arch sweep) · true-prior holdout for metric reporting · backtest window for the held-out evaluation (Step 9).

Calibration — turn rankers into forecasters SPEC · audited 2026-05-21

Map raw model scores to probabilities that mean what they claim. Critical — calibrate against the true base rate, not the case-control prior. The model is trained on the Step 4 table downsampled 1:K, so its scores reflect a ~1 % sample prior, not the real (rare) population rate. Calibrating against a slice of that 1:K pool yields probabilities wrong by 1-2 orders of magnitude ("23 %" when the truth is ~0.3 %). Fix: calibrate on a true-prior hold-out, or apply an explicit log-odds prior correction. This is the prior-correction Step 4 and Step 6 defer here — measure the true base rate, do not assume it.

Method + scope. Platt (sigmoid) default — stable on thin data; isotonic is data-hungry, use only where the calibration sample is large. Calibrate globally (matches the global model from Step 6) — per-FIPS isotonic on ~1,421 counties starves on data; split by region only if markets genuinely differ.

Acceptance — measured on the true-prior population: calibration error within a relative tolerance (absolute percentage points are the wrong frame for a rare event), reliability diagram monotonic, Brier improves. A region that fails downgrades to "ranker only — no probability claim" (score still ranks; no fake probability shown). The ±15 % in the legacy sketch is the sales model's number — set roofing's own.

Decisions to lock: measured true base rate (shared with Step 4) · roofing tolerance (value + relative vs absolute) · scope global vs per-region · method.

Output filter / sanity GATE SPEC · audited 2026-05-21

The 25K-list incident (finding 67) is fixed here, at the gate, not in the label. After calibration, apply hard rules: recent-roof exclusion (last_ROOFING_age < 15 yr), VENDOR_BLIND (property outside an INCLUDE tuple — Step 2), dedup (one mailpiece per household), confidence (combines coverage match-rate + calibration).

The gate is also a model-health detector — by design. The 15-year exclusion is intentional: if the model scores a recently-rerofed property high, the model is probably wrong. The gate catches it before the client sees it — and the catch itself is the signal. So the gate's firing rate is a monitored metric (Step 9), not silent plumbing — a rising count of high-scored recent roofs means fix the model, not the gate. A tighter very-recent band (~2-3 yr + high score) is the sharpest model-error tell, tracked separately.

Never overwrite the score. calibrated_probability is kept intact for every row; the gate only sets delivery_eligible + delivery_reason_excluded. That is what lets the recent-roof catch tell us the model is wrong — "model said 0.8, gate excluded for RECENT_ROOF". Overwriting the score to 0 / NULL would erase that evidence.

Decisions to lock: very-recent alarm band threshold + high-score cutoff · dedup unit (household, not mailing address — handle multi-property owners) · 3-level confidence definition · soft vs hard filter at the 15-yr boundary.

Deployment + monitoring SPEC · audited 2026-05-21

Value demonstration — the walk-forward backtest. The client-facing proof: stand at a past T0, train on ≤ T0, generate the list we would have delivered, then check the (T0, T0+6] outcome. "If you'd been a client in January, here's the list — of all the roofs Jan–Jun, it caught X %." Honest only with strict T0 discipline and a backtest window old enough that permit data is fully settled (permits lag — a too-recent window undercounts).

The value metric. Recall (capture rate) at a fixed list size N — recall alone is gamed by enlarging the list. The honest headline is lift over a random list of equal size: "our 15 000-property list caught 85 % of the roofs — 11× better than random." Show precision@N too (the client works the list) and a gains curve so the client picks N to capacity. Baseline = the random list — this closes the held-out-test open item; roofing does not compete against Alpha.

Ground truth is the roof permit, not the CRM. Measure recall / precision against new permit data (the label's own source). The CRM (Clients_Deals_RBB_V1) measures business conversion — a narrower, different outcome; track it separately. Avoid the feedback loop: score the full population each cycle, not just the delivered list, or keep a randomised control.

Monitoring — fast leading signals vs slow ground truth. Fast/daily: input-feature drift (PSI on features, not just score), score drift, the Step 8 gate firing rate. Slow/lagging: recall / precision vs permits — resolves ~6 months late, so it is confirmation, not a weekly trigger. Thresholds tune per-region on the calibrated probability (calibration already fixed score incomparability — no per-FIPS needed). Hybrid retraining: 90-day floor + fast-signal ceiling.

Decisions to lock: backtest window (settled permit data) · debias method (full-population scoring vs randomised control) · alert routing / on-call.