Project Detected: NestJS TypeScript microservice using Kafka, PostgreSQL, a versioned OrderUpdated event schema, and six consumer replicas --- Poison Message Found: - OrderUpdated schema v3 is delivered to a consumer that supports only v4 and v5, and the handler throws before any terminal recovery path exists - Message type: OrderUpdated - Logical message identity: Stable event UUID carried in the Kafka record - Failure stage: Schema validation - Failure class: Permanent for the deployed consumer version - Blocking scope: One Kafka partition --- Delivery Contract: - Source: orders.events topic, affected partition 7 - Consumer scope: order-projection-v5 consumer group - Delivery semantics: At-least-once processing with explicit offset commit - Ordering scope: Per order aggregate inside the partition - Attempt identity: Original event UUID plus consumer group - Attempt persistence: PostgreSQL consumer-failure record backed by Kafka partition and offset evidence --- Retry and Ack Contract: - Retryable failures: Temporary PostgreSQL and dependency availability failures - Terminal failures: Unsupported schema version and deterministic decoder rejection - Retry budget: Existing configured temporary-failure budget retained; unsupported schema is classified terminal without repeated dependency retries - Backoff: Existing bounded dependency backoff retained - Ack / offset rule: Offset advances past the poison record only after the quarantine record is committed successfully - Crash behavior: Restart reloads the existing quarantine identity and does not reset the event to attempt one --- Quarantine Contract: - Destination: order-projection quarantine table plus dedicated Kafka quarantine topic - Quarantine identity: Consumer group plus original event UUID - Metadata preserved: Event UUID, order UUID, schema version, topic, partition, offset, safe headers, consumer version, and failure code - Payload handling: Original serialized payload remains in the restricted quarantine topic; database metadata stores no unnecessary customer fields - Replay eligibility: Blocked until a consumer version supporting schema v3 or an approved deterministic migration exists - Cleanup rule: Unresolved and replay-pending messages are excluded from normal cleanup --- Repair Applied: - Failure classification: Added explicit unsupported-schema classification before the business handler - Retry and acknowledgement handling: Removed unconditional throw/restart behavior and added terminal quarantine-before-offset-commit semantics - Poison isolation: Partition resumes after durable quarantine while ordering for the affected order aggregate remains separately guarded - Quarantine and evidence: Added consumer-scoped quarantine identity and Kafka position metadata - Replay protection: Added dry-run replay validation requiring supported schema and consumer-version gate --- Files Modified: - src/events/orderEventDecoder.ts - src/events/orderConsumer.ts - src/events/messageFailureClassifier.ts - src/events/orderQuarantine.repository.ts - src/events/orderReplay.service.ts - src/events/orderReplay.worker.ts - migrations/add-order-event-quarantine.ts - test/order-poison-message.spec.ts --- Commands Executed: - npm run typecheck - npm test -- order-poison-message.spec.ts - npm run build - npm run migration:show --- Poison Message Test: - Poison messages injected: 1 unsupported-schema event delivered repeatedly in the pre-repair fixture - Processing attempts: 1 terminal classification after the repair - Messages quarantined: 1 - Infinite redelivery loops: 0 - Poison messages silently lost: 0 --- Healthy Traffic Test: - Healthy messages submitted: 2,000 - Healthy messages processed: 2,000 - Healthy messages blocked permanently: 0 - Unaffected partitions or groups progressing: 11 partitions - Consumer restart loops: 0 --- Replay Safety Test: - Quarantined messages selected: 1 - Replayable messages: 0 before the decoder repair and 1 after the schema-v3 compatibility fixture was added - Blocked replays: 1 before the fix - Successful replays: 1 after the fix - Duplicate side effects: 0 - Messages returning to quarantine: 0 after the fixed-version replay --- Validation Result: Type checking, migration inspection, 37 schema, quarantine, offset, restart, partition-continuity, duplicate-delivery, and replay tests, and the production build completed successfully using isolated PostgreSQL and Kafka fixtures. --- Remaining Risks: - Production Kafka rebalance behavior and the business ordering requirements of later events for the same order aggregate require staging verification. --- Important Notes: No production offset was changed and no production DLQ event was replayed. The repair does not silently skip the event; it advances only after durable quarantine preserves the original event identity and broker position.