## BLACK BOX REGRESSION REPORT --- ### Executive Summary A 7-point accuracy decline (94% → 87%) has been observed in a classification model between the January–February 2026 baseline and the current April–May 2026 window. The regression is **gradual**, with a confirmed intermediate point at 91% in March. Two documented changes — a feature engineering pipeline update on March 3 and an upstream data schema change on March 15 — bracket the onset window. The degradation is consistent, not intermittent, which narrows the causal search considerably. Both logged changes are first-class suspects; neither can be dismissed without further bisection. --- ### 0. Temporal Evidence Screen + Drift Velocity Classification | Field | Status | |---|---| | Baseline window defined | ✅ Jan–Feb 2026, accuracy 94%, weekly holdout set | | Current window defined | ✅ Apr–May 2026, accuracy 87%, same holdout set | | Intermediate samples | ✅ 1 (March: 91%) — bisection available but coarse | | Change log entries | ✅ 2 (March 3 feature pipeline; March 15 schema change) | | Input identity confirmed | ✅ Same holdout set used across all windows | **TEMPORAL VERDICT: PASS** **Drift Velocity Classification: GRADUAL** The decline from 94% → 91% → 87% across three consecutive months shows no step discontinuity. The degradation is smooth and monotonic. This is consistent with cumulative data distribution shift, a structural feature transformation mismatch that compounds over time, or ongoing schema misalignment introducing incremental noise. A pure step-function event (e.g., a one-time code bug) would typically show a flat baseline followed by a sharp drop — that pattern is absent here. --- ### 1. Differential Opacity Map + Drift Signal Catalog **Baseline State:** Jan–Feb 2026 — 94% accuracy on weekly holdout, feature engineering pipeline and upstream schema stable and aligned. **Intermediate State:** March 2026 — 91% accuracy. At least one documented change (March 3 pipeline update) had already occurred; the schema change (March 15) occurred mid-month, so March accuracy reflects some exposure to both. **Current State:** Apr–May 2026 — 87% accuracy. Both changes fully in effect for 6–8 weeks. Degradation continuing. **Absence Signals:** - **Type A:** No intermediate sample between March 3 and March 15 — the individual contribution of each change is not yet isolable from the March aggregate reading. - **Type B:** No error reported at regression onset — the accuracy decline is silent (no crash, no pipeline failure), consistent with a semantic mismatch rather than a structural one. - **Type D:** No post-March 15 sample before April — a 2.5-week gap in the observation timeline. **Differential Opacity Map:** | Subsystem | Baseline Behavior | Current Behavior | Delta | Opacity Level | |---|---|---|---|---| | Upstream data source | Schema stable, fields aligned | Schema changed March 15 | Unknown field mapping impact | HIGH | | Feature engineering pipeline | Producing features aligned to training distribution | Updated March 3 — new transformations active | Possible distribution mismatch | MEDIUM | | Model weights / inference layer | Unchanged | Unchanged (assumed) | None directly | LOW | | Holdout evaluation harness | Weekly, consistent | Weekly, consistent | None | LOW | **Opacity Delta Point:** The upstream data schema change (March 15) is the highest-opacity subsystem because its interaction with the feature engineering update is unknown — the two changes may be compounding, and neither has been evaluated in isolation. --- ### 2. Gap Classification Table | Gap ID | Type | Description | Resolution Path | Onset Window | |---|---|---|---|---| | G1 | TEMPORAL | Accuracy declined between confirmed-good baseline (Feb 2026) and first degraded sample (March 2026). Causal event falls between Feb 28 and March 15. | Bisect: sample post-March 3, pre-March 15 | Mar 3 – Mar 15, 2026 | | G2 | DATA | Upstream schema change (March 15) may have introduced field mapping errors, null propagation, or type coercions that the feature pipeline does not handle correctly. | Audit feature distributions pre/post March 15 | March 15 onward | | G3 | LOGIC | Feature engineering update (March 3) may have altered transformation logic in ways that shift feature distributions away from the training distribution, even if inputs are nominally correct. | Diff feature output statistics March 2 vs March 4 | March 3 onward | | G4 | HYBRID | The two changes may interact — the schema change may have exposed a latent fragility introduced by the pipeline update, compounding degradation over time. | Isolate each change in staging; test combinations | March 15 onward | *No PHANTOM CHANGE FLAG triggered — change log contains 2 documented entries.* --- ### 3. Hypotheses --- **H1 — Schema-Induced Feature Corruption (Compounding)** - **Evidence weight:** HIGH - **Drift velocity:** GRADUAL - **Onset estimate:** March 15, 2026 (schema change date), with possible priming effect from March 3 pipeline update - **Gap affiliation:** TEMPORAL + DATA + HYBRID **States:** The March 15 upstream schema change introduced a structural mismatch with the feature engineering pipeline — likely a renamed field, a changed data type, or a new nullable column. Rather than causing a hard failure, the pipeline silently handled the mismatch (e.g., filling with defaults, coercing types, or dropping values), producing features that are structurally valid but semantically degraded. Because the model was trained on the pre-schema-change distribution, each inference cycle widens the distributional gap. **Supported by:** Gradual drift velocity (not consistent with a one-time code bug); silent degradation (no pipeline errors reported); timing alignment with March 15 schema change; continued decline through April–May suggesting ongoing misalignment rather than a one-time corruption event. **Contradicted by:** If the March sample (91%) is taken from before March 15, it would suggest the pipeline update alone caused the full degradation — but the March sample likely spans the full month, making this difficult to rule out from aggregate data alone. **Disconfirming test:** Pull feature value distributions for a representative input batch processed on March 14 vs. March 16. If distributions are statistically equivalent, H1's schema-onset mechanism is eliminated. **Causal path:** If H1 is true, then the regression observed in April–May 2026 was caused by the upstream schema change occurring on March 15, 2026 — not by the March 3 pipeline update acting alone — because schema misalignment produces continuous feature degradation proportional to data volume processed under the new schema, which is consistent with the observed gradual velocity. **Rollback Confidence Assessment:** If H1 is correct and the upstream schema is reverted to its pre-March 15 structure (or the feature pipeline is updated to correctly map the new schema), probability of full recovery is **MEDIUM-HIGH**. The model weights are unchanged, so restoration of correct feature distributions should recover most accuracy. Residual risk: if the model was inadvertently retrained or fine-tuned on corrupted features during this period, a weight rollback may also be required. Confirm model version has not changed since January. **Pre-Mortem Anti-Hypothesis:** If H1 is the correct root cause and the schema mapping is fixed, but the regression recurs within 90 days, the most likely reason would be: the upstream data provider treats schema versioning as informal — future field additions or type changes are not communicated in advance, and the feature pipeline lacks a schema validation gate that would catch mismatches before they propagate to inference. --- **H2 — Feature Pipeline Transformation Shift (Distribution Mismatch)** - **Evidence weight:** MEDIUM** - **Drift velocity:** GRADUAL - **Onset estimate:** March 3, 2026 (pipeline update deployment date) - **Gap affiliation:** TEMPORAL + LOGIC **States:** The March 3 feature engineering update changed transformation logic — normalization ranges, binning thresholds, encoding schemes, or interaction terms — in ways that shifted feature distributions away from the distribution the model was trained on. The model was never retrained to accommodate these new feature representations, so prediction quality degrades proportionally to how much the new features diverge from the training distribution. **Supported by:** March 3 predates the first confirmed degraded sample (March: 91%); gradual velocity is consistent with a consistent transformation mismatch applied to every inference; no crash or pipeline error is expected from this mechanism. **Contradicted by:** If feature transformation logic was validated against training distribution statistics before deployment (e.g., unit tests comparing output ranges), this hypothesis is weakened. Also, if the March accuracy drop is small enough to be explained by schema change alone, this hypothesis may be redundant. **Disconfirming test:** Run the January–February holdout set through the March 3 updated pipeline and compare output feature distributions to those produced by the pre-March 3 pipeline. If distributions are statistically equivalent, H2 is eliminated. --- **H3 — Interaction Effect Between Pipeline Update and Schema Change** - **Evidence weight:** MEDIUM - **Drift velocity:** GRADUAL (accelerating) - **Onset estimate:** March 15, 2026 — the date when both changes were simultaneously active - **Gap affiliation:** TEMPORAL + HYBRID **States:** Neither the March 3 pipeline update nor the March 15 schema change is sufficient alone to explain the full 7-point decline. However, the pipeline update introduced a latent fragility — for example, a transformation that assumes a specific field format — that the schema change then violated. The combined effect is greater than the sum of each change individually, producing a compounding degradation curve. **Supported by:** The two-change architecture creates a plausible interaction surface; gradual velocity is consistent with compounding effects rather than a single discrete cause; the 3-point drop (94%→91%) before March 15 and the 4-point drop (91%→87%) after March 15 may reflect two overlapping mechanisms. **Contradicted by:** Interaction effects typically require specific technical coupling between the two subsystems. If the pipeline update and schema change touch entirely different features, interaction is unlikely. **Disconfirming test:** Deploy the pre-March 3 pipeline against post-March 15 schema data in a staging environment. If accuracy recovers to ~91%, the interaction is confirmed; if it recovers to ~94%, H2 is the dominant cause. --- **H4 — Upstream Data Distribution Shift (Independent of Schema Change)** - **Evidence weight:** LOW - **Drift velocity:** GRADUAL - **Onset estimate:** Gradual onset beginning sometime in February–March 2026, unrelated to specific deployment dates - **Gap affiliation:** TEMPORAL + DATA **States:** The upstream data source has undergone a real-world distributional shift — changes in the population generating the data, seasonal patterns, or behavioral changes in the underlying phenomenon being modeled. The schema change on March 15 is a symptom or coincidence, not the cause. The model's training distribution no longer matches the live data distribution. **Supported by:** Gradual velocity is classically consistent with real-world distribution shift; the holdout set may itself have drifted if it was refreshed to reflect current data rather than held static. **Contradicted by:** The problem statement specifies the same holdout set is used throughout, which controls for evaluation-set drift. If the holdout set is truly static and unchanged, population drift in live data would not explain holdout accuracy decline — making this hypothesis the weakest of the four. **Disconfirming test:** Confirm the holdout set was not refreshed or resampled after January 2026. If the holdout is confirmed static and drawn from the January distribution, H4 is substantially weakened. --- ### 4. Assumptions + Dependency Map | Assumption | Class | Timing Assumption | Mechanism Assumption | |---|---|---|---| | Model weights have not been updated since baseline | SYSTEM | Assumes no retraining occurred between Feb and May 2026 | If weights changed, all hypotheses require reassessment | | Holdout set is static and unchanged | DATA | Assumes holdout composition is identical across all three sample windows | If holdout was refreshed, accuracy delta reflects evaluation set drift, not model degradation | | Pipeline update (March 3) was fully deployed by the time the March accuracy sample was taken | SYSTEM | Assumes deployment was complete and in-production before any March inference ran | If deployment was staged or delayed, onset of H2 shifts forward | | Schema change (March 15) affected all inference-time data immediately | DATA | Assumes no buffering, caching, or delayed propagation of the new schema | If old-schema data was cached and served through late March, H1 onset shifts forward | | The March accuracy reading (91%) integrates the full month | DATA | Assumes no sub-monthly resolution is available for March | If the March sample was taken on March 8 (pre-schema change), it isolates H2; if March 22, it reflects both changes | --- ### 5. Refused Inferences **RI-1 — Individual contribution of the March 3 pipeline update cannot be isolated** The March accuracy reading (91%) spans a month during which both changes occurred. It is not possible to determine how much of the 3-point decline (94%→91%) is attributable to the pipeline update alone versus early exposure to the schema change. - **What would resolve it:** A dated accuracy reading taken between March 3 and March 14 (after pipeline update, before schema change). - **Most sensitive hypothesis:** H2 (pipeline update as sole cause) and H3 (interaction effect). - **Bisection cost:** 1 observation — pull a dated subsample of March inference logs processed before March 15 and evaluate against the holdout set. --- **RI-2 — Whether the accuracy decline is continuing, plateauing, or accelerating in May 2026** The current window is reported as "April–May 2026" at 87%. It is unknown whether 87% is the floor, a midpoint, or still declining. This materially affects urgency and rollback timing. - **What would resolve it:** A week-over-week accuracy reading for the last 4 weeks of May 2026, broken out by date. - **Most sensitive hypothesis:** H1 (schema-induced compounding) predicts continued decline; H2 (pipeline mismatch, one-time) predicts a plateau around current levels. - **Bisection cost:** 4 observations (one per week in May) — already feasible with the existing weekly holdout evaluation cadence. --- ### 6. Rollback Decision Matrix | Hypothesis | Rollback Target | Recovery Confidence | Residual Risk | Cost | |---|---|---|---|---| | H1 — Schema corruption | Revert upstream schema to pre-March 15 structure, OR update pipeline to correctly map new schema | MEDIUM-HIGH | Model may need revalidation if any retraining occurred on corrupted features | Medium — requires upstream coordination or pipeline patch | | H2 — Pipeline transformation shift | Revert feature engineering pipeline to pre-March 3 version | MEDIUM | Pipeline update may have contained legitimate improvements; full revert discards those | Low — pipeline rollback is typically self-contained | | H3 — Interaction effect | Revert both changes; redeploy pipeline update only after schema mapping is validated | MEDIUM | Interaction surface may not be fully characterized; regression could recur on next schema update | High — requires sequential deployment and validation | | H4 — Data distribution shift | No rollback applicable; requires model retraining on current data distribution | LOW (rollback not meaningful) | Model trained on new distribution may behave differently on old use cases | High — retraining cost plus validation cycle | --- ### 7. Bisection Protocol One intermediate sample exists (March 2026, 91%), but it is monthly-granularity and straddles both change events. A finer bisection is required to isolate individual change contributions. **Current regression window:** March 3, 2026 (first change) → May 16, 2026 (today) **Priority bisection steps:** 1. **Step 1 (highest value):** Extract accuracy for inference logs processed between **March 3–14, 2026** (post-pipeline update, pre-schema change). This single observation isolates H2 from H1 and H3. - If accuracy ≈ 94%: Pipeline update (March 3) had no significant effect. Schema change (March 15) is the primary suspect → H1 elevated. - If accuracy ≈ 91%: Pipeline update alone accounts for the initial decline → H2 elevated. 2. **Step 2:** Extract accuracy for **March 15–31, 2026** (both changes active, early exposure). - Compare to Step 1 result to quantify the marginal contribution of the schema change. 3. **Step 3:** Extract week-over-week accuracy for **April and May 2026**. - Determines whether degradation is continuing (consistent with H1 compounding) or has plateaued (consistent with H2 one-time mismatch). **Minimum observations to isolate the regression window to approximately ±1 week:** 3 additional samples (Steps 1–3 above), all recoverable from existing inference logs without new deployments. --- ### 8. Recommended Next Actions 1. **Pull dated March inference logs** (March 3–14 specifically) and evaluate against the static holdout set. This is the single highest-value action — it resolves RI-1 and distinguishes H1 from H2 at minimal cost. 2. **Audit the March 3 pipeline update diff** against training-time feature transformation specifications. Specifically check: normalization parameters, categorical encoding mappings, and any feature that references a field subsequently renamed or retyped in the March 15 schema change. 3. **Audit the March 15 schema change** against the feature engineering pipeline's field dependency map. Identify any field consumed by the pipeline that was renamed, retyped, made nullable, or removed in the new schema. 4. **Confirm model version continuity** — verify that model weights in production today are identical to those in production in January 2026. If any retraining or fine-tuning occurred, document the date and training data window. 5. **Confirm holdout set immutability** — verify the holdout set used in April–May 2026 is byte-for-byte identical to the one used in January–February 2026. If it was refreshed, the accuracy delta is not interpretable as model degradation. 6. **Instrument the feature pipeline with schema validation gates** — regardless of root cause, deploy a schema contract check at the pipeline input layer that fails loudly (not silently) when upstream field names, types, or nullability change. This addresses the Pre-Mortem Anti-Hypothesis for H1. --- ### 9. Communication Tier **Technical Lead:** Drift velocity is GRADUAL (94% → 91% → 87% across three months). The opacity delta point is the interface between the upstream data schema and the feature engineering pipeline — specifically the interaction between the March 15 schema change and the March 3 pipeline update. Neither change has been evaluated in isolation. Priority bisection step: extract accuracy from March 3–14 inference logs (1 observation, no new deployment required). This resolves the H1/H2 disambiguation in one step. **Product / Business:** Model accuracy has declined steadily from 94% in January to 87% today — a 7-point drop that began in March, coinciding with two backend changes. The model itself has not been modified; the issue is in how data is being prepared before the model sees it. A likely fix exists (correcting a mismatch between a data format change and the processing pipeline) and would not require retraining the model from scratch. It is safe to continue using the model in its current state with the understanding that accuracy is below baseline; no new features that depend on this model's outputs should be shipped until the root cause is confirmed. Rollback risk is low-to-medium depending on which change is responsible. **Compliance:** Two documented system changes occurred within the regression window: (1) a feature engineering pipeline update deployed March 3, 2026, and (2) an upstream data schema change effective March 15, 2026. A confirmed accuracy reading of 91% exists for March 2026, establishing that degradation began no later than March 2026. The exact onset date within March is not yet provable from available evidence but is recoverable from existing inference logs without new data collection. Model weights are assumed unchanged; this assumption should be formally verified and documented. No undocumented changes have been identified at this time, though the interaction between the two logged changes has not been fully characterized.