**ASSUMPTIONS:** Both inputs are usable. The orchestrator/deploy mechanism, current and target ioredis versions, Redis server version, BullMQ version, and base image are not named, so those are UNKNOWN or `<VERIFY>`. ## 1. CHANGE SUMMARY | Unit | Changes | Unchanged | |---|---|---| | U1 Node runtime 18→22 | Runtime in API + 2 BullMQ workers, Docker base image, CI build | PostgreSQL 14, app logic | | U2 ioredis major bump | Redis client API/defaults, used by API and (via BullMQ) workers | Redis server | | U3 bcrypt rebuild | Native binary rebuilt against Node 22 ABI | Hash format on stored passwords | Three independent risk units are bundled. Touches auth, cache/queue traffic, container builds, and live traffic at ~2k req/min. ## 2. CHANGE SURFACE MAP | Surface | Exposure | Why | Tag | |---|---|---|---| | Queues (BullMQ) | HIGH | BullMQ depends on ioredis; a client major bump can alter connection/option semantics | HIGHLY LIKELY (BullMQ is built on ioredis) | | Auth (bcrypt) | HIGH | Native module must match Node 22 ABI; failure blocks login | HIGHLY LIKELY | | Infrastructure/Docker | HIGH | Base image and toolchain change; native build needs compiler/prebuilt binary | HIGHLY LIKELY | | Dependencies (transitive) | MED | Packages may break on Node 22 (engines, removed APIs) | PLAUSIBLE | | Cache | MED | ioredis behavior changes affect cache reads/writes | HIGHLY LIKELY | | CI/CD | MED | Actions `setup-node`/Docker build steps pinned to 18 | PLAUSIBLE | | Database | LOW | pg driver behavior on Node 22 unverified | UNKNOWN | ## 3. DEPENDENCY IMPACT GRAPH - Node 22 → Express, TypeScript build target, pg driver, all transitive deps (VERIFY: `npm ls` engines/peer warnings) - ioredis → API cache code AND BullMQ (shared client dependency; BullMQ's required ioredis range is `VERIFY: BullMQ package.json peerDependencies/dependencies`) - bcrypt → depends on Node-API prebuilds or node-gyp toolchain in the image (`VERIFY: bcrypt release notes for Node 22 prebuilt support`) - Hidden coupling: the API and both workers likely share one `package.json`/image, so one bad bump breaks all three processes (PLAUSIBLE; UNKNOWN whether images are separate). ## 4. CONTRACT COMPATIBILITY MATRIX | Contract | Class | Mechanism / requirement | |---|---|---| | HTTP API responses | UNCHANGED | No API change intended (CONFIRMED by inputs) | | Password hashes in PG | UNCHANGED | bcrypt hash format is stable; verify old hashes verify under new build (HIGHLY LIKELY) | | Redis key/data formats | UNKNOWN | Depends on ioredis serialization/option changes; VERIFY changelog | | BullMQ job payloads/queue keys | CONDITIONALLY COMPATIBLE | Old-version workers and new-version workers may share queues during rollout; VERIFY BullMQ/ioredis compat | | Worker↔API job schema | UNKNOWN | Not described in inputs | ## 5. BREAKAGE PATHS | # | Change → Immediate break → Secondary → Impact | Tag | Reason | |---|---|---|---| | 1 | bcrypt binary mismatched/missing on Node 22 → process crashes at import or hash/compare throws → login/registration fail → auth outage | HIGHLY LIKELY | Native modules are ABI/toolchain bound | | 2 | ioredis major changes connection/retry/option defaults → reconnect storms or command errors → cache misses hit PG, BullMQ stalls | PLAUSIBLE | Major bumps commonly alter defaults; VERIFY changelog | | 3 | Mixed old/new workers on same queues → lock/stalled-job handling differs → duplicate or stuck jobs | PLAUSIBLE | Shared Redis state across versions | | 4 | Node 22 behavior changes (fetch/undici, module resolution, removed/deprecated APIs) in a dependency → runtime exceptions on specific routes → partial 5xx | PLAUSIBLE | 40% coverage leaves untested paths | | 5 | Docker build succeeds on CI but native rebuild differs from prod arch/libc → runtime-only crash | PLAUSIBLE | Build vs run environment mismatch; UNKNOWN target arch | | 6 | Perf/memory profile shift under Node 22 GC/defaults → latency or OOM at 2k req/min | PLAUSIBLE | No baseline in inputs | ## 6. FAILURE PROPAGATION MAP - Path 1: **SYSTEM-WIDE** for auth (every authenticated request), because one shared image/runtime. - Path 2/3: **CROSS-COMPONENT** (shared Redis fans out to API cache, PG load via misses, and both workers). - Path 4: **CONTAINED** to affected routes (UNKNOWN which). - Path 5/6: **UNKNOWN** until measured. ## 7. CHANGE COUPLING ANALYSIS | Unit | Deploy separately | Validate separately | Roll back separately | |---|---|---|---| | U1 Node 22 | Yes (image tag) | Yes | Yes (previous image) | | U2 ioredis | Yes, if bcrypt/Node held constant | Yes | Yes, until queue state diverges (VERIFY) | | U3 bcrypt | Coupled to U1 (ABI) | Yes, via unit test | With U1 | **Recommendation:** ship U1+U3 first (Node 22, ioredis unchanged), then U2 alone. Do not bundle. ## 8. PRECONDITION REGISTER | Precondition | Status | |---|---| | Rollback artifact (previous image tag retained) | UNKNOWN | | Test/staging environment mirroring prod | UNKNOWN (no canary exists) → **BLOCKING** | | Dependency compatibility with Node 22 (bcrypt, pg, BullMQ, others) | MISSING | | ioredis changelog and BullMQ compat reviewed | MISSING | | Integration coverage on auth + queue paths | MISSING (40% overall) → **BLOCKING** for auth/queue | | Observability baselines (error rate, latency, queue depth) | UNKNOWN | | Ability to route a fraction of traffic | UNKNOWN → **BLOCKING** for canary | | Feature flag / config parity | UNKNOWN | ## 9. TEST & VALIDATION MATRIX | Test | Risk | Pass | Fail → Next | |---|---|---|---| | `npm ci` + build on Node 22 in CI | Path 4/5 | Clean install, no engine errors | Fix or pin dep | | bcrypt hash+compare against existing prod-format hashes | Path 1 | Old hashes verify | Rebuild/upgrade bcrypt | | Integration suite on Node 22 image | Path 4 | Same pass set as Node 18 | Triage diffs | | BullMQ enqueue→process→retry→fail on new ioredis | Path 2/3 | Job lifecycle identical | Hold U2 | | Mixed-version worker test (old+new on same queue) | Path 3 | No duplicates/stalls | Drain queue instead of mixing | | Load test at ~2k req/min | Path 6 | Latency/memory within baseline | Investigate | ## 10. SAFE ROLLOUT STRATEGY No canary exists, so create the smallest substitute: **stage 0** staging/shadow environment (or a single extra instance receiving limited traffic if the platform allows: UNKNOWN), then **rolling** with a hold at a small fraction. Two workers allow **worker-first** promotion: run one worker on the new image, watch queue behavior, then the other. Ship U1+U3, hold, then U2 as a separate release. ## 11. ROLLBACK MAP | Stage | Method | Limitation | |---|---|---| | Node 22 API/worker instances | Redeploy previous image tag | Needs retained tag (UNKNOWN) | | ioredis (U2) | Redeploy previous image | **Unsafe** once new client has written incompatible queue/lock state (VERIFY) | | bcrypt | Reverts with image | Hashes created remain valid (HIGHLY LIKELY) | ## 12. OBSERVABILITY REQUIRED | Signal | Why | Stop condition | |---|---|---| | HTTP 5xx rate, per route | Path 1, 4 | `SET FROM BASELINE: 5xx rate` | | Login success/failure rate | Path 1 | `SET FROM BASELINE: login failures` | | p95/p99 latency | Path 6 | `SET FROM BASELINE: p95` | | BullMQ waiting/active/failed/stalled counts | Path 2, 3 | `SET FROM BASELINE: failed and stalled` | | Redis connection errors/reconnects | Path 2 | `SET FROM BASELINE: reconnects` | | Container memory/restarts | Path 5, 6 | `SET FROM BASELINE: RSS, restarts` | ## 13. RELEASE BLOCKERS 1. No production-like environment or traffic-splitting mechanism (BLOCKING). 2. Auth and queue paths lack proven integration coverage at 40% overall (BLOCKING). 3. Dependency compatibility with Node 22 and ioredis/BullMQ compat unverified. 4. No baselines for the stop conditions. 5. Rollback image retention and queue-state compatibility unconfirmed. **Gate classification: READY WITH BLOCKERS** (safe to prepare; not safe to expose). ## 14. EXECUTABLE ARTIFACTS **Artifact A: Runbook (U1+U3 first, then U2 as a separate repeat)** ```markdown PREPARE 1. CMD: record current prod image tag and Node/ioredis/bullmq/bcrypt versions: `<VERIFY: your deploy mechanism> ` + `npm ls ioredis bullmq bcrypt` EXPECT: previous image tag saved and still pullable STOP IF: previous tag cannot be pulled → go to Artifact D / do not proceed 2. CMD: capture baselines (5xx, login failures, p95, queue counts, Redis reconnects, RSS) for `<VERIFY: baseline window>` EXPECT: numbers recorded for Artifact C STOP IF: any signal has no data source BUILD 3. CMD: change Node to 22 in Dockerfile base image and GitHub Actions setup-node (`<VERIFY: files/lines pinning Node 18>`); keep ioredis unchanged EXPECT: diff limited to Node version pins 4. CMD: `npm ci && npm rebuild bcrypt --build-from-source` inside the Node 22 image (`<VERIFY: bcrypt install method used>`) EXPECT: exit 0, bcrypt loads under Node 22 STOP IF: build/import error 5. CMD: `npm run build && npm test` (`<VERIFY: your script names>`) EXPECT: same passing set as Node 18 run STOP IF: new failures VERIFY 6. CMD: in the new image run a script that hashes and compares a password and verifies a known existing hash from staging/test data EXPECT: both succeed STOP IF: throws or mismatch 7. CMD: run integration suite + BullMQ lifecycle test against a non-production Redis EXPECT: enqueue/process/retry/fail all behave as on Node 18 STOP IF: stalled/duplicate jobs EXPOSE 8. CMD: deploy new image to ONE worker only (`<VERIFY: your deploy mechanism>`) EXPECT: worker consumes jobs, no restarts for the hold period STOP IF: crash loop, failed/stalled counts rise vs baseline 9. CMD: deploy to a minimal API slice/one instance if routing allows (`<VERIFY: traffic-splitting capability>`) EXPECT: 5xx, login failure, p95 within baseline STOP IF: any Artifact C abort condition PROMOTE 10. CMD: roll remaining worker, then API in steps per Artifact C EXPECT: signals stable at each hold 11. CMD: after hold, repeat steps 3-11 for the ioredis bump only (Node held constant) EXPECT: as above; queue state checked before each step ``` **Artifact B: Pre-flight checklist** ```markdown - { } BLOCKING: A production-like test env or traffic-limited slice exists. Prove: run smoke test against it. Status: UNKNOWN - { } BLOCKING: Auth + queue paths have integration tests passing on Node 22. Prove: CI run link. Status: MISSING - { } BLOCKING: A mechanism exists to send only a fraction of traffic to the new version. Prove: routing config `<VERIFY: your deploy mechanism>`. Status: UNKNOWN - { } Previous image tag retained and pullable. Prove: pull it. Status: UNKNOWN - { } Dependencies checked for Node 22 support (bcrypt, pg, bullmq, others). Prove: `npm ls`, release notes. Status: MISSING - { } ioredis major changelog read; BullMQ compat confirmed. Prove: `<VERIFY: ioredis changelog, BullMQ docs>`. Status: MISSING - { } Baselines captured for all Section 12 signals. Prove: recorded numbers. Status: UNKNOWN - { } Config/env parity between staging and prod. Prove: diff of env. Status: UNKNOWN ``` **Artifact C: Rollout & abort card** ```markdown | STAGE | EXPOSURE | WATCH SIGNAL | HOLD MINUTES | ABORT IF | NEXT STAGE ONLY IF | |---|---|---|---|---|---| | 0 Staging | none | tests, load test | n/a | any regression vs Node 18 | all Section 9 tests pass | | 1 One worker | jobs on 1 of 2 workers | failed/stalled counts, restarts | SET FROM BASELINE: queue behavior | failed/stalled above baseline, crash loop | stable for hold | | 2 Minimal API slice | small traffic fraction (if possible) | 5xx, login failures, p95 | SET FROM BASELINE: p95 | any signal beyond baseline | stable for hold | | 3 Second worker | all jobs | queue counts, Redis reconnects | SET FROM BASELINE: reconnects | rising stalled/duplicates | stable for hold | | 4 Remaining API | full traffic | all signals, RSS | SET FROM BASELINE: RSS/restarts | any stop condition | stable for hold; then start ioredis release | ``` **Artifact D: Rollback procedure** ```markdown 1. Stop promotion; freeze further deploys. 2. Redeploy previous image tag to the API: `<VERIFY: your deploy mechanism>`. EXPECT: 5xx and login failures return to baseline. 3. Redeploy previous image to workers, one at a time. EXPECT: queue counts normalize. 4. For the ioredis release only: before rollback, inspect queue/lock keys for new-format state `<VERIFY: ioredis/BullMQ changelog>`; drain or pause queues first if formats differ. =================== POINT OF NO RETURN =================== For U2: once the new ioredis client has written queue or lock state the old client cannot read (UNKNOWN; VERIFY), redeploying the old image may not recover in-flight jobs. Jobs may need manual replay. Not undoable after this point: in-flight job state written by the new client, and any jobs processed twice. For U1/U3: rollback stays safe, since bcrypt hashes and PG data are unchanged (HIGHLY LIKELY). ``` ## 15. CHANGE-SAFETY EXECUTION PACK **SAFE TO PREPARE:** Node 22 Dockerfile/CI changes, bcrypt rebuild, dependency audit, test additions, baseline capture (no production exposure). **BLOCKED:** Any production exposure until a traffic-limited slice or staging environment exists and auth/queue tests pass. **FIRST CHANGE:** Node 22 + bcrypt rebuild only, ioredis untouched. **FIRST TEST:** bcrypt hash/compare (including existing-format hashes) inside the Node 22 image, then the BullMQ lifecycle test. **FIRST RELEASE STEP:** One worker on the new image. **WATCH:** BullMQ failed/stalled counts and login failure rate. **ABORT IF:** Crash loop, auth failures, or stalled/duplicate jobs beyond baseline (`SET FROM BASELINE`). **ROLLBACK:** Redeploy previous image tag; safe for U1/U3. For U2, limited by queue/lock state written by the new client (VERIFY). **POINT OF NO RETURN:** First time the new ioredis client writes queue or lock state the old client cannot read. **DONE WHEN:** All stages held at baseline, ioredis released separately with the same evidence, and no queue anomalies across a full hold period. **STATUS:** READY WITH BLOCKERS This is decision support and a prepared runbook. Nothing here has been executed or tested, and it doesn't replace review by the engineers who own the system.