Project Detected: NestJS TypeScript services using PostgreSQL, TypeORM, transactional outbox records, Kafka, and a billing consumer group --- Event Path Reviewed: - Event: OrderCreated - Event identity: Stable outbox event UUID - Correlation identity: Checkout operation UUID - Producer: Order service - Broker path: orders.events topic, partitioned by order UUID, consumed by billing-v4 - Consumer: Billing service - Expected destination: Billing order projection and payment-intent eligibility state --- Loss Symptom: - Missing effect: 17 confirmed orders have no billing projection - First observed gap: Kafka consumer progression - Delay versus loss: Confirmed checkpoint skip rather than ordinary consumer lag - Duplicate or stale evidence: Several surrounding events were processed successfully, and no duplicate suppression record exists for the missing IDs --- Evidence Chain: - Producer evidence: All 17 order mutations and matching outbox rows committed in the same PostgreSQL transaction - Publication evidence: Outbox rows contain confirmed Kafka partition and offset metadata - Broker evidence: All 17 event IDs are present in the disposable reproduction topic - Consumer evidence: Billing logs show the containing batch was fetched, but no per-event inbox rows exist for the missing records - Destination evidence: Billing projection rows are absent - Ack / checkpoint evidence: The consumer commits the final offset of a 100-record batch even when parallel handlers have not all committed --- Loss Boundary: - Last confirmed stage: Kafka delivered the batch to the billing consumer - First missing stage: Durable billing processing before committed consumer checkpoint - Failure class: Batch checkpoint advanced beyond unfinished records - Confidence: Authoritative --- Recovery Contract: - Replay safety: Safe after adding a transactional billing inbox and stable payment-operation identities - Stable replay identity: Original OrderCreated event UUID - Retry scope: Missing event IDs only - DLQ behavior: Permanent schema or business failures move to billing-order-dlq with original event metadata - Reconciliation method: Compare committed OrderCreated outbox IDs against billing inbox IDs by bounded order range --- Repair Applied: - Producer or outbox protection: Existing transactional outbox retained; publication evidence now records Kafka acknowledgement metadata consistently - Publication and broker diagnostics: Added event ID, partition, offset, and correlation metadata to structured tracing - Consumer acknowledgement and checkpointing: Replaced end-of-batch commit with contiguous durable-offset tracking after inbox and projection commit - Gap detection and reconciliation: Added bounded order-outbox versus billing-inbox reconciliation - Replay protection: Added transactional inbox uniqueness and targeted replay by original event IDs --- Files Modified: - apps/orders/src/outbox/orderOutboxPublisher.ts - apps/billing/src/events/orderConsumer.ts - apps/billing/src/events/consumerCheckpoint.ts - apps/billing/src/events/eventInbox.entity.ts - apps/billing/src/reconciliation/orderEventGapScanner.ts - migrations/add-billing-event-inbox.ts - test/order-event-loss.spec.ts --- Commands Executed: - npm run typecheck - npm test -- order-event-loss.spec.ts - npm run build - npm run migration:show --- Loss Reproduction: - Injected failure: Process terminated while handlers for offsets 1542-1550 were still committing but the batch-final offset was eligible for commit - Event reached broker: Yes - Event reached consumer: Yes - Destination mutation committed: No for the interrupted records - Event acknowledged or checkpointed: Yes under the old implementation --- Replay Test: - Events selected: 17 - Events replayed: 17 - Duplicate local effects: 0 - Duplicate external effects: 0 - Unresolved events: 0 --- End-to-End Verification: - Events produced: 2,000 - Events broker-confirmed: 2,000 - Events consumed: 2,000 - Destination effects verified: 2,000 - Unexplained gaps: 0 --- Validation Result: Type checking, migration inspection, 36 publication, checkpoint, crash, duplicate, gap-reconciliation, and replay tests, and the production build completed successfully using isolated PostgreSQL and Kafka fixtures. --- Remaining Risks: - Production Kafka rebalance timing and the billing consumer’s real partition concurrency require staging verification under representative load. --- Important Notes: No production offsets were reset and no production events were replayed. Recovery targets only verified missing event IDs and preserves their original logical identities.