Skill Details
Runtime
Any model
Example user message
[userMessage]: Our Node/TypeScript worker uses one circuit breaker around a partner API, but each Kubernetes replica has independent state. The breaker opens after failures and waits 20 seconds, then half-open allows every waiting worker to probe at once. We also have an SDK retry inside the breaker and a queue retry outside it. After the partner recovers, the probe burst sometimes knocks it back down. Repair the repository locally with fake clocks and a scripted dependency. Preserve existing threshold values. Do not call production, change provider limits, or decide that breaker state must be globally shared.
Example agent response
# 1. Repair Decision
**Decision: HALF-OPEN PROBE STAMPEDE AND RETRY/BREAKER OWNERSHIP DRIFT FOUND — SAFE LOCAL PATCH AVAILABLE; shared-vs-local production state remains an architecture decision.**
# 2. Scope & Safety Boundary
Reviewed: breaker wrapper, partner SDK retry config, queue retry, worker concurrency, breaker metrics, tests, and docs.
Not accessed: production partner API, Kubernetes, live queue, provider limits, or production breaker state. Existing threshold and 20-second open interval are preserved.
# 3. Breaker Source Register
| Component | Current behavior | Finding |
|---|---|---|
| breaker | one per process | local state |
| SDK | retries internally | nested attempt owner |
| queue | retries job | second retry owner |
| half-open | all waiting work may probe | recovery stampede |
| failure classifier | transport + timeout | needs exact review |
| metrics | open/closed only | probe pressure hidden |
# 4. State Machine Reconstruction
Closed → supplied failure threshold reached → Open for existing 20s → Half-Open.
Current Half-Open behavior lets many workers resume together. SDK retries can multiply each probe, while queue retries may create another wave.
# 5. Drift Findings
Confirmed:
- unbounded half-open probe concurrency;
- nested retry ownership;
- recovery surge;
- per-instance state not documented as local.
Not proven:
- local breaker state is incorrect;
- production should use distributed breaker state;
- supplied thresholds are wrong.
# 6. Canonical Breaker Contract
- breaker remains owned by the partner client wrapper;
- supplied thresholds/open interval remain unchanged;
- half-open admits a small explicit probe set;
- ordinary calls fail/defer through existing policy while probes run;
- SDK retry is disabled or narrowed for breaker-owned attempts where repository semantics permit;
- queue retry remains the durable retry owner for failed jobs;
- cancellation is not counted as partner failure;
- unsafe writes are not blindly replayed.
# 7. Repair Applied
Prepared:
- half-open probe gate;
- single transition owner per breaker instance;
- fake-clock-safe reset timer;
- retry ownership normalization;
- failure-classification tests;
- metrics for probe admitted/rejected, nested retries prevented, state transitions, and recovery duration.
# 8. Retry / Half-Open Safety
A half-open probe is one logical dependency attempt, not a burst of SDK retries. Queue replay happens only through the existing durable job contract.
The patch does not introduce shared breaker state across replicas.
# 9. Deterministic State Test Matrix
1. Existing threshold opens breaker.
2. Fake clock reaches existing 20s boundary.
3. Half-open admits only configured local probe concurrency.
4. 100 waiting jobs do not become 100 dependency calls.
5. Successful probe set closes breaker.
6. Failed probe reopens breaker.
7. Stale reset timer cannot close a newly reopened breaker.
8. Cancellation does not increment dependency failure count.
9. SDK + queue retry does not multiply one probe unexpectedly.
10. State remains explicitly per-process in local tests.
# 10. Observability & Rollout
Track breaker state/generation, transition reason, half-open admitted/rejected, dependency attempts, queue retries, SDK retries suppressed, fallback/defer count, and recovery duration.
# 11. Files Modified
- `src/partner/partnerBreaker.ts`
- `src/partner/partnerClient.ts`
- `src/jobs/partnerJob.ts`
- `src/metrics/breakerMetrics.ts`
- `tests/partnerBreaker.test.ts`
- `docs/partner-breaker-contract.md`
# 12. Commands Executed
- `npm test -- partnerBreaker`
- `npm run typecheck`
- `npm run lint`
# 13. Validation
Local results: half-open gate PASS; fake-clock transitions PASS; stale timer fencing PASS; nested retry control PASS; cancellation classification PASS; recovery probe behavior PASS; typecheck/lint PASS.
Not verified: production cross-replica behavior, provider recovery capacity, or whether shared breaker state is desirable.
# 14. Remaining Decisions & Handoff
Owner decisions: permitted half-open probe concurrency, shared-vs-local breaker architecture, and production rollout alert thresholds.
**Handoff: PATCH READY FOR LOCAL REVIEW — HALF-OPEN PROBES, RETRY OWNERSHIP, AND TIMER FENCING VERIFIED LOCALLY; PRODUCTION CAPACITY AND CROSS-INSTANCE STATE POLICY NOT VERIFIED.**
Suggested commit: `fix(resilience): bound breaker half-open probes`
Or get it with Select10 Select prompts/mo — under $2 each
Learn more
Instant access
Commercial use
Money‑backBy purchasing this skill, you agree to our terms of service
3
Views
Repair circuit-breaker state drift across clients, workers, services, and instances. This skill finds duplicate breaker ownership, inconsistent failure classification, nested retry amplification, unbounded half-open probes, reset-timer races, stale shared state, and fallback blind spots; then prepares bounded repository patches, deterministic fake-clock state tests, recovery controls, and observability without calling production dependencies or inventing thresholds.
...more
Added 1 week ago
