1) Quick check (90 seconds overview) Consistency check: The workflow, error location and frequency agree. Reading record_id from a CSV containing id can raise KeyError before validation or insertion. “Every current fixture import” describes the reported failure pattern; only one fixture job was evaluated. Worker-attempt counts and outcomes were not supplied. Observed symptom: KeyError: record_id at header mapping. Supplied change history: The exporter renamed record_id to id; the importer still expects record_id. Evidence limitation: The supplied “logs” describe the failure but contain no raw traceback or attempt history. Three priority cause categories: - Schema-contract mismatch: Directly explains the reported failure. - Retry-control mismatch: Automatic retries are enabled, conflicting with mandatory operator approval before every replay after failure. - Transaction/checkpoint consistency: Commit and checkpoint are separate operations, creating a potential replay window. This is a latent risk, not evidence of the current failure. 2) Hypothesis list (prioritized) Probability labels are qualitative diagnostic judgments, not measured statistics. H1. Stale header lookup causes KeyError. Probability: high. Impact: high. Reason: Explicitly supported by the supplied rename and symptom. H2. Missing header preflight allows an incompatible CSV to reach record processing. Probability: high. Impact: medium. Reason: The reported result is a field-access exception rather than a controlled schema rejection. H3. Retry configuration can replay failed work without operator approval. Probability: high. Impact: high. Reason: Enabled retries are supplied facts; whether any replay occurred is unknown. H4. A successful commit followed by checkpoint failure can cause replay of committed records. Probability: medium. Impact: high. Reason: These operations are separate; no such failure was reported. H5. Plain INSERT encounters primary-key violations when replay reaches existing record_id values. Probability: medium. Impact: high. Reason: The primary key prevents duplicate stored IDs, but INSERT alone does not provide safe replay behavior. H6. A duplicate or invalid record aborts a transaction containing otherwise valid rows. Probability: medium. Impact: high. Reason: Transaction boundaries and row-level error handling were not supplied. H7. Existing IDs with different labels lack an explicit operator-review path. Probability: medium. Impact: high. Reason: Plain INSERT rejects a collision but does not itself classify it for review. H8. Invalid-row reporting is incomplete when processing exits on an exception. Probability: medium. Impact: medium. Reason: A reporting step exists, but its failure-path behavior is unspecified. H9. Fetch failures, particularly HTTP 401, enter the generic retry path. Probability: low. Impact: high. Reason: Fetch handling is unspecified; enabled retries warrant checking the mandatory stop-on-401 rule. 3) Test plan / tests All tests below are proposed and have not been executed. Every replay after failure requires operator approval, including fixture tests. H1 — Header mismatch: - Inspect the raw CSV header and traceback from the evaluated job. - Compare the exporter’s field names with the importer’s lookup. - With operator approval, reproduce using a minimal id,label fixture containing fictional data. - Confirm whether the exception occurs before any database write. - Exclude this hypothesis only if the actual failing CSV contains the expected key and the traceback identifies another access. H2 — Header preflight: - Inspect parsing and validation code. - Test separate fixtures with id,label; missing ID header; missing label header; and conflicting id and record_id headers. - Verify unsupported or ambiguous headers produce a controlled diagnostic before database writes. - Required evidence: parsed field names, validation result and before/after database contents. H3 — Replay approval: - Inspect worker retry settings and any outer launcher or scheduler retries. - Trigger a controlled fixture failure. - Verify the job stops and cannot start another worker attempt without recorded operator approval. - Required evidence: job identifier, distinct attempt identifiers, timestamps, retry configuration and approval record. - One job may have several attempts; count them separately. Additional binding checks: - Inject failure after commit but before checkpoint; approve replay and verify existing rows remain intact. - Test identical-ID/identical-label and identical-ID/different-label cases. The latter must require operator review without overwrite. - Mix valid and invalid rows; verify valid rows are preserved and invalid rows are listed. - Simulate HTTP 401; verify immediate stop without automatic retry. - Compare the final count report against committed database state. 4) Root cause analysis The most likely immediate cause is the importer’s stale record_id header lookup after the exporter changed the field name to id. Supporting evidence: - The rename is explicitly supplied. - The importer still reads record_id. - The reported KeyError names that exact field. - The reported failure location is header mapping. This is strongly supported by the brief, but raw fixture inspection and traceback verification remain outstanding. Automatic retries are a separate confirmed configuration conflict. Transaction, checkpoint and conflict-handling concerns are diagnostic hypotheses or latent risks; they are not established causes of this KeyError. 5) Fix plan Implement within the existing Python worker and PostgreSQL database, without vendor migration or schema changes. Ordered actions: - Disable automatic retries at every launch layer. Require operator approval before every replay after failure. - Add header preflight and explicitly map exporter id to internal record_id before validation. - Reject missing or ambiguous mappings before writes; do not silently choose between conflicting headers. - Validate IDs and nonempty labels, retaining row numbers and reasons for invalid-row reporting. - Retain the primary key as the uniqueness guarantee. Use conflict-safe insertion without updating existing labels, then classify collisions: identical values are already stored; differing labels remain unchanged and enter operator review. - Ensure anticipated invalid rows and collisions do not discard otherwise valid records. Keep unexpected database failures visible. - Write the checkpoint only after commit. Make approved replay safe if checkpoint writing fails. - Produce counts for newly inserted, already present, invalid and review-required records. - Enforce immediate stop on HTTP 401, with no automatic replay. Quick fix: - Disable retries, correct the explicit header mapping, add preflight validation and verify the immediate failure with a small fixture. Clean fix: - Complete conflict classification, valid-row preservation, failure reporting and commit/checkpoint replay handling before treating replay as safe. Four-hour initial allocation: - 30 minutes: inspect evidence and disable retries. - 45 minutes: header mapping and preflight. - 90 minutes: conflict handling and row outcomes. - 75 minutes: targeted failure tests and reporting. Use local fictional fixtures; no paid service is required. Keep fixture spending within EUR 50. If the full scope exceeds four hours, leave replay approval enforced and report unfinished checks. Risks: - Mapping both headers without ambiguity checks could select the wrong identifier. - Silently ignoring conflicts could hide differing labels. - Advancing the checkpoint before commit could skip uncommitted records. - Per-row database operations may reduce throughput; fixture testing should assess the impact. 6) Prevention Five measures: - Validate CSV headers before processing, so exporter changes fail clearly. - Maintain a small exporter/importer contract fixture covering the accepted header mapping. - Record job IDs, attempt IDs, failure stage and replay approval, so retries remain auditable. - Monitor commit/checkpoint failures and reconcile approved replays against database state. - Test mandatory controls: invalid-row listing, uniqueness, no overwrite without operator review, stop on 401 and approval before replay. Mini checklist for future changes: - Does the exporter still match the importer’s accepted headers? - Are valid rows preserved and invalid rows listed? - Can each record_id exist at most once? - Are differing existing labels unchanged pending operator review? - Does HTTP 401 stop processing? - Does every replay after failure require operator approval? - Are counts based on committed outcomes? - Have proposed checks actually been executed and their results recorded?