# 1. PRODUCTION SYSTEM MAP **TRIGGERS** - Scheduled poll of Zendesk for unresolved tickets (CONFIRMED: "continuously monitors") - Possible webhook-driven trigger on ticket update — PLAUSIBLE, UNKNOWN which mechanism is actually used **AGENT / ORCHESTRATOR** - Central reasoning loop that evaluates each ticket, decides escalation-worthiness, drafts briefs, and sequences tool calls. Orchestration framework (single agent vs. multi-step pipeline) — UNKNOWN. **MODEL CALLS** - Ticket triage/classification (escalate vs. not) - Evidence synthesis into an escalation brief - Possibly a routing decision (which engineering team) — HIGHLY LIKELY given "routes them to engineering teams" **RETRIEVAL** - Ticket history from Zendesk - Product documentation from internal knowledge base - Retrieval freshness mechanism (index age, cache invalidation) — UNKNOWN **TOOLS** - Zendesk (read ticket state/history, possibly write status updates) - Knowledge base (read) - Jira (create issue, update issue) - Slack (notify/route to engineering teams, request approval) - Durable task state store (read/write) **EXTERNAL SYSTEMS** - Zendesk, Jira, Slack, knowledge base platform — all third-party, all independent failure domains **STATE** - Durable task state per ticket/case (CONFIRMED: "durable task state" is a named component) - Must persist across multi-day follow-up (CONFIRMED risk: "losing multi-day task state") **MEMORY** - Long-lived task/case memory implied by follow-up tracking — HIGHLY LIKELY; no separate long-term agent memory system specified — UNKNOWN **QUEUES** - Implied queue of unresolved tickets awaiting evaluation — HIGHLY LIKELY; explicit queue infrastructure UNKNOWN **HUMAN GATES** - Approval required for "consequential customer actions" (CONFIRMED). Exact definition of "consequential" — UNKNOWN - Approval channel likely Slack — PLAUSIBLE **ACTIONS** - Create Jira issue - Route/notify engineering team - Possibly update/respond to customer ticket — PLAUSIBLE, gated by approval **VERIFICATION** - Confirming Jira issue was actually created before reporting success — not specified; must be designed (see Stage 2) - Confirming no duplicate issue exists for same ticket — not specified; must be designed **COMPLETION** - Case marked resolved/escalated-and-tracked once Jira issue is created and follow-up loop closes — inferred structurally **RECOVERY PATHS** - Not specified — UNKNOWN. Must be designed from scratch (Stage 15). Missing topology (explicit UNKNOWNs): trigger mechanism, orchestration architecture, retrieval freshness policy, queue infra, exact approval scope, verification steps, recovery mechanism. --- # 2. PRODUCTION INVARIANTS **INV-1: NO TASK SILENTLY DISAPPEARS** - WHY: A case with unresolved engineering need that vanishes from tracking directly harms customers and violates the stated goal. - VIOLATION SIGNAL: Ticket present in Zendesk as unresolved but absent from durable task state, or task state shows no update for longer than expected working cadence. - DETECTION SOURCE: Reconciliation job comparing Zendesk open-ticket set against task-state case set. - OPERATIONAL CONSEQUENCE: Customer issue stalls indefinitely with no engineering visibility. - RESPONSE: Re-enqueue missing case; alert if reconciliation finds a gap. **INV-2: NO DUPLICATE EXTERNAL SIDE EFFECT (Jira issue)** - WHY: Explicit stated risk ("creating duplicate Jira issues"); duplicates waste engineering time and break routing. - VIOLATION SIGNAL: More than one open Jira issue linked to the same ticket ID. - DETECTION SOURCE: Pre-creation check against task state's stored Jira-issue reference; post-creation reconciliation against Jira API by ticket-ID tag/field. - OPERATIONAL CONSEQUENCE: Engineering confusion, wasted triage, possible conflicting fixes. - RESPONSE: Block creation if a linked issue already exists and is open; alert on detected duplicates for manual merge. **INV-3: NO CONSEQUENTIAL ACTION WITHOUT APPROVAL** - WHY: Explicit stated risk ("bypassing approval gates"). - VIOLATION SIGNAL: A customer-facing or otherwise consequential action recorded as executed with no corresponding approval record in task state. - DETECTION SOURCE: Action log cross-referenced against approval log at write time (hard gate) and in audit (soft check). - OPERATIONAL CONSEQUENCE: Unauthorized customer-impacting action; potential trust/compliance issue. - RESPONSE: Treat as CRITICAL CONTAINMENT — pause the acting capability pending investigation. **INV-4: NO STALE EVIDENCE USED BEYOND ITS VALIDITY BOUNDARY** - WHY: Explicit stated risk ("using stale evidence"); an escalation brief built on outdated ticket history or docs misleads engineering. - VIOLATION SIGNAL: Evidence retrieved timestamp older than a defined freshness boundary — BASELINE REQUIRED for what "stale" means here (no TTL supplied). - DETECTION SOURCE: Timestamp comparison recorded at retrieval time, stored in trace. - OPERATIONAL CONSEQUENCE: Escalation brief may misstate current ticket state, wasting engineering time or missing a resolution that already happened. - RESPONSE: Re-fetch evidence before brief finalization; block brief completion on stale critical fields (e.g., ticket status). **INV-5: NO UNBOUNDED RETRY ON FAILED TOOLS** - WHY: Explicit stated risk ("repeatedly calling failed tools"). - VIOLATION SIGNAL: Same tool call (same case, same tool, same parameters) exceeds a retry count within a time window — THRESHOLD REQUIRES OPERATIONAL BASELINE. - DETECTION SOURCE: Per-case, per-tool retry counter in task state. - OPERATIONAL CONSEQUENCE: Wasted API quota/rate-limit exhaustion, delayed case progress, possible cascading tool degradation classification. - RESPONSE: Circuit-break that tool for that case; escalate to human queue; do not block unrelated cases unless the tool is globally failing. **INV-6: NO CROSS-TASK STATE CONTAMINATION** - WHY: Multi-day, multi-case state persistence creates risk of one case's evidence/context leaking into another's brief. - VIOLATION SIGNAL: Brief or Jira issue references ticket IDs, customer data, or evidence not belonging to the case's own state record. - DETECTION SOURCE: Schema/ID validation at write time; audit sampling of generated briefs against source case ID. - OPERATIONAL CONSEQUENCE: Wrong customer data in wrong Jira issue — potential privacy and correctness failure. - RESPONSE: CRITICAL CONTAINMENT — halt brief generation pipeline, audit recent outputs. **INV-7: NO SUCCESS REPORTED AFTER FAILED REQUIRED ACTION** - WHY: If Jira creation fails but the agent marks the case as "escalated," the case is effectively lost while appearing handled. - VIOLATION SIGNAL: Task state marked complete/escalated with no corresponding verified Jira issue ID. - DETECTION SOURCE: Post-action verification step (fetch-back check) required before state transition to "escalated." - OPERATIONAL CONSEQUENCE: Same as INV-1 — silent loss, but harder to detect because it looks resolved. - RESPONSE: Do not permit state transition without verification; on verification failure, revert to pending and alert. **INV-8: NO UNBOUNDED AGENT LOOP** - WHY: Iterative reasoning (re-evaluating a ticket, re-drafting a brief) could loop without termination. - VIOLATION SIGNAL: Same case processed through N reasoning/tool cycles without state advancement — THRESHOLD REQUIRES OPERATIONAL BASELINE. - DETECTION SOURCE: Step counter per case per run. - OPERATIONAL CONSEQUENCE: Resource waste, delayed case, potential runaway cost. - RESPONSE: Force case into human review queue after threshold. **INV-9: NO COMPLETION WITH REQUIRED WORK UNRESOLVED** - WHY: A case shouldn't be marked "handled" while follow-up tracking is still open. - VIOLATION SIGNAL: Case closed in task state while linked Jira issue remains open/unresolved with no scheduled follow-up check. - DETECTION SOURCE: Cross-check task-state completion flag against Jira issue status. - OPERATIONAL CONSEQUENCE: Follow-up tracking goal (explicitly required) is defeated. - RESPONSE: Block closure; keep in follow-up-tracking state until Jira resolves or human overrides. --- # 3. FAILURE TAXONOMY **MODEL** - Incorrect escalation decision (escalates non-issues, or fails to escalate real ones) - Unsupported brief content (claims not grounded in retrieved evidence) - Wrong team routing decision - Instruction failure (ignores approval requirement in its own reasoning trace) **RETRIEVAL** - Missing ticket history (Zendesk API partial response) - Stale knowledge-base article (doc updated after retrieval, agent uses old version) - Wrong source (retrieves docs for wrong product/module) - Retrieval outage (KB or Zendesk search unavailable) **TOOL** - Zendesk timeout/rate limit - Jira create-issue permission failure - Jira invalid parameters (bad project key, missing required field) - Partial success (Jira issue created but Slack notification fails — inconsistent downstream state) - Slack outage preventing approval request delivery **WORKFLOW** - Wrong routing (case sent to wrong engineering team) - Missing step (brief created but Jira creation step skipped) - Premature completion (case marked done before follow-up loop closes — INV-9) - Orchestration failure (case dropped between pipeline stages) **STATE / MEMORY** - Lost state (multi-day case state not persisted/restored — explicit stated risk) - Stale state (state reflects pre-update ticket status) - Corruption (partial write leaves inconsistent record) - Incorrect resumption (resumes case from wrong checkpoint after restart) - Cross-task contamination (INV-6) **ACTION** - Duplicate Jira issue (explicit stated risk) - Wrong target (issue created in wrong Jira project/team) - Unverified mutation (Jira issue creation not confirmed) - Partial execution (issue created, but linkage back to Zendesk ticket not written) **HUMAN GATE** - Approval bypass (explicit stated risk) - Approval deadlock (approver never responds, case stuck) - Human non-response beyond SLA — BASELINE REQUIRED for what SLA is - Incorrect escalation (routed for approval when not actually consequential, or vice versa) **RECOVERY** - Retry storm (INV-5 violation) - Failed compensation (attempted rollback of a bad Jira issue fails) - Repeated dead end (case keeps hitting same tool failure across restarts) - Resume failure (case cannot resume after worker crash) **INFRASTRUCTURE** - Queue backlog (unresolved-ticket poll produces more cases than pipeline can process) - Worker failure (process handling a case dies mid-run) - Network failure to any external system - Resource saturation — UNKNOWN whether concurrency limits exist **BUSINESS OUTCOME** - Escalation created but engineering never acts because brief lacked necessary evidence (technically complete, operationally useless) - Customer issue resolved by support before engineering starts work, but Jira issue remains open (wasted engineering cycle) --- # 4. SIGNAL ARCHITECTURE | SIGNAL | SOURCE | WHAT IT INDICATES | WHAT IT DOES NOT PROVE | DECISION ENABLED | |---|---|---|---|---| | Case age since last state transition | Task state | Possible stuck case | Doesn't prove cause (tool vs. model vs. human) | Trigger stuck-task investigation | | Tool error rate per tool | Tool call logs | Tool degradation | Doesn't prove root cause is the tool itself vs. bad params | Circuit-break candidate | | Repeated identical tool call count | Tool call logs keyed by case+tool+params | Retry loop | Doesn't prove infinite — could be legitimate scheduled retry | Enforce INV-5 | | Duplicate Jira-issue-per-ticket count | Jira reconciliation job | INV-2 violation occurring | Doesn't prove cause (race condition vs. missing check) | Immediate containment + audit | | Evidence retrieval timestamp vs. brief finalization timestamp | Trace | Possible stale evidence | Doesn't prove the stale evidence changed the outcome | Block or re-fetch before finalization | | Approval wait duration | Task state / Slack event log | Possible approval deadlock | Doesn't prove approver unavailable vs. genuinely reviewing | Escalate reminder, then reassign | | Action-without-approval-record count | Action log vs. approval log join | INV-3 violation | Doesn't prove intent — could be logging bug | CRITICAL CONTAINMENT regardless, then investigate | | Reconciliation gap (Zendesk open vs. task-state tracked) | Reconciliation job | INV-1 violation | Doesn't prove agent's fault — could be Zendesk sync lag | Re-enqueue + alert if persistent | | Escalation rate (per unit time) | Task state completion events | Possible over/under-escalation behavior drift | Doesn't prove which direction is wrong without baseline | Quality drift investigation (Stage 9) | | Case step count without state advancement | Orchestrator loop counter | Possible unbounded loop (INV-8) | Doesn't prove infinite vs. legitimately complex case | Force human review | --- # 5. AGENT TRACE SPECIFICATION | FIELD | PURPOSE | DIAGNOSTIC VALUE | SENSITIVE-DATA CONCERN | |---|---|---|---| | RUN ID | Unique identifier per execution pass | Correlates all events in one run | None | | TASK ID (case ID / Zendesk ticket ID) | Ties trace to specific customer case | Enables per-case reconstruction | Ticket ID alone low risk; avoid embedding customer PII in the ID | | TENANT BOUNDARY | If multi-tenant, isolates data | Prevents cross-tenant leakage diagnosis | Must not leak across tenants in shared dashboards | | TRIGGER | What initiated this run (poll, webhook, retry) | Distinguishes scheduled vs. reactive vs. recovery runs | None | | STATE VERSION | Version/checkpoint of task state read at start | Detects stale-state or concurrent-write issues | None | | MODEL STEP | Each model invocation with role (triage, brief-draft, routing) | Reconstructs reasoning sequence | Must not log full customer message bodies unnecessarily — log references, not raw PII, where feasible | | CONTEXT / SOURCE REFERENCES | Which ticket history and KB docs were retrieved, with timestamps | Enables staleness and grounding checks (INV-4) | Store references/IDs, not full document content, where possible | | DECISION | Escalate/no-escalate, routing target | Core business logic checkpoint | None | | TOOL CALL | Tool name, parameters, attempt number | Diagnoses tool failures and retry patterns | Redact credentials/tokens from parameters | | TOOL RESULT | Success/failure, response summary | Confirms whether action actually succeeded | Avoid storing full raw payloads containing customer PII beyond need | | ACTION | Jira issue created, Slack message sent, etc. | Ties side effects to a specific run | None beyond above | | APPROVAL | Approver ID, decision, timestamp | Verifies INV-3 compliance | Approver identity is internal, low sensitivity, but still access-controlled | | RETRY | Count and reason | Detects retry storms (INV-5) | None | | ESCALATION | To which team, with what evidence bundle reference | Diagnoses routing correctness | None | | VERIFICATION | Post-action confirmation result | Confirms INV-7 compliance | None | | COMPLETION | Final state and closure reason | Confirms INV-9 compliance | None | | ERROR | Error type, tool/model source | Root-cause input | Avoid raw stack traces containing secrets | | RECOVERY | What recovery path was taken, from what checkpoint | Confirms resume correctness | None | Do not log full customer conversation text, credentials, or Jira/Slack tokens in the trace; reference by ID and fetch on demand when a human investigates. --- # 6. MULTI-DIMENSION HEALTH MODEL **SYSTEM HEALTH** - HEALTHY: All external tool connections reachable; queue depth within BASELINE REQUIRED range. - DEGRADATION: Intermittent tool errors below circuit-break threshold. - FAILURE: One or more tools unreachable beyond retry policy. - EVIDENCE SOURCE: Tool call success/failure logs. - RESPONSE: Enter tool-specific degradation mode (Stage 12). **WORKFLOW HEALTH** - HEALTHY: Cases progress through expected state transitions within expected cadence — BASELINE REQUIRED. - DEGRADATION: Rising count of cases without state advancement. - FAILURE: Reconciliation shows cases missing from task state (INV-1). - EVIDENCE SOURCE: Task-state transition log + reconciliation job. - RESPONSE: Stuck-task investigation; re-enqueue missing cases. **AGENT BEHAVIOR HEALTH** - HEALTHY: Escalation decisions and brief quality stable relative to baseline — BASELINE REQUIRED (no accuracy metric supplied). - DEGRADATION: Rise in unsupported-brief flags or escalation-rate deviation. - FAILURE: Sustained pattern of incorrect escalation confirmed by engineering feedback (false-positive/negative rate). - EVIDENCE SOURCE: Brief content audits, engineering feedback on escalation validity. - RESPONSE: Quality drift investigation (Stage 9); consider reduced-autonomy mode. **TOOL HEALTH** - HEALTHY: Success rate and latency within normal range per tool. - DEGRADATION: Elevated error rate or latency, still functional. - FAILURE: Tool unreachable or consistently erroring. - EVIDENCE SOURCE: Per-tool call logs. - RESPONSE: Circuit-break specific tool (Stage 8/14). **STATE HEALTH** - HEALTHY: State reads/writes succeed; no version conflicts. - DEGRADATION: Occasional version conflicts or slow writes. - FAILURE: State loss or corruption detected (INV-6, multi-day persistence failure). - EVIDENCE SOURCE: State store write confirmations, schema validation errors. - RESPONSE: Freeze writes to affected case(s), CRITICAL CONTAINMENT if widespread. **HUMAN-GATE HEALTH** - HEALTHY: Approvals resolved within expected window — BASELINE REQUIRED. - DEGRADATION: Wait times trending up. - FAILURE: Approval deadlock (no response beyond SLA) or approval bypass detected (INV-3). - EVIDENCE SOURCE: Approval wait duration, action-vs-approval log join. - RESPONSE: Reminder/reassignment for deadlock; CRITICAL CONTAINMENT for bypass. **OUTCOME HEALTH** - HEALTHY: Escalated cases result in engineering action and eventual resolution — BASELINE REQUIRED. - DEGRADATION: Rising rate of escalations with no engineering action taken. - FAILURE: Pattern of escalations that are ignored, duplicated, or resolved outside the tracked Jira issue. - EVIDENCE SOURCE: Jira issue status correlated with case closure. - RESPONSE: Business-outcome investigation (Stage 9), review brief quality and routing logic. --- # 7. STUCK / LOOP / SILENT FAILURE DETECTION | CONDITION | DETECTION LOGIC | REQUIRED STATE | ALERT | AUTOMATED CONTAINMENT | HUMAN ESCALATION | |---|---|---|---|---|---| | STUCK TASK | No state transition for case beyond expected cadence (BASELINE REQUIRED) | Last-transition timestamp per case | WARNING → ACTION REQUIRED if sustained | None automatic; flag for reprocessing | Ops reviews case after threshold | | ABANDONED TASK | Case present in Zendesk as unresolved but absent from task state (INV-1) | Reconciliation diff | ACTION REQUIRED | Re-enqueue automatically | Ops notified if re-enqueue also fails | | REPEATED TOOL CALL | Same tool+params+case exceeds N attempts in window | Per-case tool-call counter | WARNING then PAUSE CANDIDATE for that case | Circuit-break tool for that case | Route case to human queue | | REPEATED REASONING STEP | Same model-step type repeats beyond N cycles without decision change | Per-case step counter + decision hash | WARNING | Force case to human review | Ops/eng reviews reasoning trace | | RETRY STORM | Aggregate retry count across cases for one tool spikes beyond baseline | Global per-tool retry counter | ACTION REQUIRED → PAUSE CANDIDATE | Circuit-break tool globally | Notify on-call | | QUEUE STARVATION | Queue depth grows while processing rate flatlines | Queue depth + throughput metrics | ACTION REQUIRED | None automatic (needs infra diagnosis) | On-call infra review | | APPROVAL DEADLOCK | Approval pending beyond SLA (BASELINE REQUIRED) | Approval-request timestamp | WARNING → ACTION REQUIRED | Auto-reminder to approver | Reassign to backup approver | | STATE NOT ADVANCING | Case state version unchanged across multiple processing attempts | State version history | WARNING | Flag for manual state inspection | Ops reviews | | FALSE COMPLETION | Case marked complete but linked Jira issue open/unverified (INV-7, INV-9) | Cross-check job | ACTION REQUIRED | Revert case to pending state | Ops audits verification logic | | SILENT TOOL FAILURE | Tool call logged as success but downstream verification fails (e.g., Jira issue not actually retrievable) | Post-call verification fetch | ACTION REQUIRED | Retry with verification, else flag | Ops/eng investigates tool integration | --- # 8. TOOL & DEPENDENCY OBSERVABILITY **ZENDESK** - PURPOSE: Retrieve ticket list, ticket history, update ticket status - SUCCESS: Expected fields returned / write acknowledged - FAILURE: Error response, timeout, auth failure - LATENCY: Response time — BASELINE REQUIRED - RATE-LIMIT: 429 or documented limit headers — UNKNOWN if Zendesk plan limits are known - PARTIAL-SUCCESS RISK: Ticket list returned but individual ticket history fetch fails - RETRY POLICY: Bounded retry with backoff; do not retry auth failures — THRESHOLD REQUIRES OPERATIONAL BASELINE - FALLBACK: Use last-known state with staleness flag (bounded by INV-4) - CIRCUIT-BREAK: Sustained failure rate beyond baseline → pause new case intake from Zendesk **KNOWLEDGE BASE** - PURPOSE: Retrieve product documentation for evidence - SUCCESS: Relevant doc(s) returned - FAILURE: Search outage, empty result for known topic - LATENCY: BASELINE REQUIRED - RATE-LIMIT: UNKNOWN - PARTIAL-SUCCESS RISK: Returns outdated doc version silently - RETRY POLICY: Bounded retry - FALLBACK: Proceed with brief flagged "documentation unavailable" rather than fabricating grounding — do not allow model to substitute unsupported claims - CIRCUIT-BREAK: Sustained outage → hold brief generation, queue cases for later finalization **JIRA** - PURPOSE: Create/update escalation issues - SUCCESS: Issue ID returned and confirmed retrievable - FAILURE: Permission error, invalid project/field, timeout - LATENCY: BASELINE REQUIRED - RATE-LIMIT: UNKNOWN - PARTIAL-SUCCESS RISK: Issue created but linkage field to Zendesk ticket not saved (INV-2/INV-9 risk) - RETRY POLICY: Idempotency key required (e.g., ticket ID as dedup key) before any retry — do not blindly retry create calls - FALLBACK: Hold case in "pending escalation" state; do not proceed to Slack notification until verified - CIRCUIT-BREAK: Sustained failure → PAUSE CANDIDATE for all new Jira-issue-creation actions; existing approvals held **SLACK** - PURPOSE: Route notifications, request/receive human approval - SUCCESS: Message delivered, approval response received - FAILURE: Delivery failure, no response - LATENCY: BASELINE REQUIRED - RATE-LIMIT: UNKNOWN - PARTIAL-SUCCESS RISK: Notification sent but approval mechanism (button/thread) fails silently, leaving case stuck - RETRY POLICY: Bounded retry on delivery; do not retry approval prompts indefinitely — force reassignment instead - FALLBACK: Secondary notification channel — UNKNOWN if one exists - CIRCUIT-BREAK: Sustained Slack outage → HUMAN-APPROVAL-ONLY becomes impossible, so force PAUSED mode for consequential actions **DURABLE TASK STATE STORE** - PURPOSE: Persist case progress across multi-day lifecycle - SUCCESS: Write acknowledged with version increment - FAILURE: Write rejected, version conflict, read returns stale/missing record - LATENCY: BASELINE REQUIRED - RATE-LIMIT: N/A typically - PARTIAL-SUCCESS RISK: Write partially applied (field-level inconsistency) - RETRY POLICY: Retry idempotent reads freely; writes require conflict detection before retry - FALLBACK: None — this is the source of truth; failure here is CRITICAL - CIRCUIT-BREAK: Any sustained state-store failure → PAUSE entire agent (this is the one dependency with no safe degraded mode) --- # 9. QUALITY DRIFT MODEL | DIMENSION | QUALITY SIGNAL | BASELINE NEEDED | POSSIBLE CAUSES | INVESTIGATION | RELEASE RELATIONSHIP | |---|---|---|---|---|---| | GROUNDING | Rate of brief claims not traceable to retrieved evidence | Yes — none supplied | Prompt change, retrieval failure, model version change | Sample briefs, check citations against source references | Check against Stage 10 change log | | ROUTING | Rate of Jira issues reassigned to a different team post-creation | Yes | Routing logic error, KB gap on team ownership, model drift | Compare routing decision vs. eventual correct team | Correlate with prompt/workflow version | | TOOL SELECTION | Rate of unnecessary or missing tool calls per case | Yes | Prompt/tool-schema change | Review tool-call sequences vs. expected pattern | Correlate with tool schema/version changes | | TASK COMPLETION | Completion rate and time-to-completion trend | Yes | Any upstream failure category | Check failure taxonomy breakdown for the period | Correlate with all version types | | ESCALATION | Escalation rate deviation (up or down) from baseline | Yes | Over-cautious or under-cautious model behavior, KB changes | Compare against historical rate, sample false positives/negatives | Correlate with model/prompt version | | HUMAN OVERRIDE | Rate at which humans reject/modify agent's proposed action | Yes | Declining decision quality | Review overridden cases for pattern | Correlate with model version | | RECOVERY | Rate of successful vs. failed resume attempts | Yes | State/recovery logic bug | Review recovery trace for failed resumes | Correlate with workflow/state-schema version | | DUPLICATE PREVENTION | Rate of near-duplicate Jira issues caught vs. missed | Yes | Dedup-check logic gap or race condition | Audit dedup-check hit/miss log | Correlate with workflow version | | BUSINESS OUTCOME | Rate of escalations leading to actual engineering resolution | Yes | Poor brief quality, wrong routing, low-value escalations | Cross-reference Jira resolution against case history | Correlate with all version types | Do not attribute any of the above to "the model" without first checking Stage 10 change correlation — retrieval, workflow, and tool changes are equally plausible causes. --- # 10. CHANGE CORRELATION MODEL Track and timestamp: - MODEL VERSION (triage/brief/routing model versions, if distinct) - PROMPT VERSION (per prompt role) - WORKFLOW VERSION (orchestration logic) - TOOL VERSION (Zendesk/Jira/Slack/KB API or integration version) - RETRIEVAL INDEX / SOURCE VERSION (KB index build/version) - POLICY VERSION (approval/consequential-action definitions) - CONFIGURATION (thresholds, retry limits, circuit-break settings) - DEPLOYMENT (orchestrator/service deploy timestamps) For any observed regression (e.g., escalation-rate spike, grounding-rate drop): overlay the regression's onset time against this change log. A change within the relevant lookback window is a **correlation candidate**, not a confirmed cause. Confirmation requires isolating the change (e.g., rollback test, A/B) — DESIGNED — NOT EXECUTED by default. --- # 11. ALERT ARCHITECTURE | ALERT | LEVEL | WHAT HAPPENED | WHY IT MATTERS | WHO ACTS | FIRST CHECK | |---|---|---|---|---|---| | Reconciliation gap detected | ACTION REQUIRED | Case(s) unresolved in Zendesk but missing from task state | Direct INV-1 violation — case may be silently lost | On-call ops | Check reconciliation job output for affected ticket IDs | | Duplicate Jira issue detected | ACTION REQUIRED / CRITICAL if recurring | Two+ open issues linked to same ticket | INV-2 violation — engineering confusion | Eng + ops | Check dedup-check log for the case; determine if check was bypassed or raced | | Action without approval record | CRITICAL CONTAINMENT | Consequential action executed with no matching approval entry | INV-3 violation — unauthorized action | On-call ops + eng lead | Freeze consequential-action capability; pull full trace for the run | | Sustained tool failure (any) | WARNING → PAUSE CANDIDATE | Tool error rate exceeds baseline for sustained window | Tool health degrading; risk of retry storm or stuck cases | On-call eng | Check tool status page / recent tool-side changes | | Retry storm | PAUSE CANDIDATE | Global retry count for one tool spikes | Risk of external rate-limit ban, wasted resources | On-call eng | Identify tool, check if circuit-break already engaged | | Approval deadlock | WARNING → ACTION REQUIRED | Approval pending beyond SLA | Cases stall, customer-facing delay | Ops | Check approver availability, reassign | | State store write failure | CRITICAL CONTAINMENT | Durable state write rejected/failed | No safe degraded mode for source of truth | On-call eng (highest priority) | Check state store health directly | | False completion detected | ACTION REQUIRED | Case closed without verified Jira issue | INV-7/INV-9 violation, looks resolved but isn't | Ops | Revert case, inspect verification step logs | | Escalation-rate deviation | WARNING | Escalation rate outside historical baseline | Possible quality drift (over/under escalation) | Eng/product | Run Stage 9 investigation, check Stage 10 change log | | Business-outcome decline | WARNING | Escalations not resulting in engineering resolution | Agent "succeeding" without producing value | Eng/product leadership | Sample recent briefs for quality, check routing accuracy | Do not alert on every individual tool failure — only sustained-degradation, invariant-violation, accumulated-stuck-work, or outcome-level signals, per instruction. --- # 12. DEGRADATION MODES **NORMAL** - ENTRY: All tools healthy, invariants holding. - ALLOWED: Full autonomous operation within approval gates. - BLOCKED: None beyond standing approval requirement. - USER-VISIBLE STATE: Cases process normally. - EXIT: Any dependency degrades beyond baseline. **REDUCED AUTONOMY** - ENTRY: Quality drift signal (e.g., escalation-rate deviation, grounding-rate drop) without a hard tool failure. - ALLOWED: Draft briefs and recommendations; propose routing. - BLOCKED: Autonomous Jira creation — require human confirmation even for previously-auto-approved cases. - USER-VISIBLE STATE: Ops sees "reduced autonomy" flag on dashboard. - EXIT: Quality signal returns to baseline over a defined observation window — BASELINE REQUIRED. **READ-ONLY** - ENTRY: Jira or Slack (write paths) failing while Zendesk/KB (read paths) remain healthy. - ALLOWED: Continue retrieval, evaluation, brief drafting, queuing. - BLOCKED: Any write action (Jira create, Slack notify). - USER-VISIBLE STATE: Cases accumulate in "ready to escalate, pending write capability" state. - EXIT: Write-path tool recovers and passes health check. **HUMAN-APPROVAL-ONLY** - ENTRY: Consequential-action classification uncertainty rises, or a near-miss on INV-3 is detected. - ALLOWED: All actions proceed only after explicit human approval, including ones normally auto-approved. - BLOCKED: Any autonomous action. - USER-VISIBLE STATE: Approval queue volume increases; ops notified of mode change. - EXIT: Root cause of uncertainty resolved and verified by ops/eng. **QUEUE / DEFER** - ENTRY: Queue backlog or worker saturation. - ALLOWED: Continue processing at reduced rate; prioritize oldest/highest-severity cases. - BLOCKED: New low-priority case intake, until backlog clears. - USER-VISIBLE STATE: Dashboard shows queue depth trend and prioritization in effect. - EXIT: Backlog returns below threshold — BASELINE REQUIRED. **FALLBACK WORKFLOW** - ENTRY: Knowledge base unavailable but escalation still time-sensitive. - ALLOWED: Escalate with ticket-history-only evidence, explicitly flagged "documentation unavailable." - BLOCKED: Claims that require doc-grounding. - USER-VISIBLE STATE: Brief marked with an evidence-gap flag for engineering. - EXIT: KB access restored. **PAUSED** - ENTRY: State-store failure, confirmed approval bypass, confirmed cross-task contamination, or sustained multi-tool failure. - ALLOWED: No new case intake or actions; in-flight verification-only tasks may complete if safe. - BLOCKED: Everything else. - USER-VISIBLE STATE: Agent shows PAUSED status; ops notified immediately. - EXIT: Root cause fixed, manual review completed, explicit human resume. --- # 13. INCIDENT DIAGNOSTIC TREES **SYMPTOM: Completion rate drops** → FIRST DISCRIMINATING SIGNAL: Is queue depth also rising? → POSSIBLE CAUSES: Queue backlog/worker saturation (if yes) vs. tool failure or approval deadlock (if no) → NEXT CHECK: If no queue rise, check per-tool error rates, then approval-wait durations → ISOLATED FAILURE DOMAIN: Infrastructure vs. specific tool vs. human-gate → CONTAINMENT: QUEUE/DEFER mode or circuit-break specific tool or reminder/reassign approver → RECOVERY: Scale workers / restore tool / clear approval backlog **SYMPTOM: Tasks become stuck** → FIRST DISCRIMINATING SIGNAL: Where is the last recorded state transition (which stage)? → POSSIBLE CAUSES: Stuck at retrieval (KB/Zendesk issue), stuck at tool call (repeated failure), stuck at approval (deadlock) → NEXT CHECK: Pull trace for stuck case(s), inspect last tool call and its result → ISOLATED FAILURE DOMAIN: Determined by trace stage → CONTAINMENT: Circuit-break the specific stalled tool, or reassign approval → RECOVERY: Resume from last verified checkpoint after root cause fixed **SYMPTOM: Duplicate actions increase** → FIRST DISCRIMINATING SIGNAL: Are duplicates concentrated on one case (race condition) or spread across many (systemic dedup-check failure)? → POSSIBLE CAUSES: Concurrent processing of same case (race), dedup-check logic bug, dedup-check bypassed after a deploy → NEXT CHECK: Check Stage 10 change log for recent workflow/dedup-logic deploys → ISOLATED FAILURE DOMAIN: Workflow/dedup-check logic → CONTAINMENT: Disable auto Jira-creation (READ-ONLY or HUMAN-APPROVAL-ONLY) until fixed → RECOVERY: Merge duplicate issues manually, patch dedup logic, replay affected cases **SYMPTOM: Tool failures spike** → FIRST DISCRIMINATING SIGNAL: Single tool or multiple? → POSSIBLE CAUSES: Single tool → external outage/API change; multiple → shared infra issue (network, credentials expiry) → NEXT CHECK: Check tool provider status page; check recent credential/config changes → ISOLATED FAILURE DOMAIN: Specific external dependency, or shared infra → CONTAINMENT: Circuit-break affected tool(s); if state store affected, PAUSE entirely → RECOVERY: Wait for external recovery or rotate credentials; resume with verification pass over affected cases **SYMPTOM: Human escalations increase** → FIRST DISCRIMINATING SIGNAL: Is this approval-request volume, or agent-initiated "needs human review" volume (loop/uncertainty triggers)? → POSSIBLE CAUSES: Genuine rise in consequential cases vs. INV-8 loop threshold triggering more often (model uncertainty rising) → NEXT CHECK: Compare against Stage 10 change log; sample flagged cases for legitimacy → ISOLATED FAILURE DOMAIN: Model behavior vs. genuine case-mix shift → CONTAINMENT: REDUCED AUTONOMY if model-driven → RECOVERY: Prompt/model fix and re-baseline, or accept as genuine demand shift **SYMPTOM: Unsupported outputs increase** → FIRST DISCRIMINATING SIGNAL: Correlated with a KB/retrieval outage or a model/prompt change? → POSSIBLE CAUSES: Retrieval degradation (fallback workflow producing evidence-gap briefs) vs. model grounding failure → NEXT CHECK: Check retrieval success rate for the same period → ISOLATED FAILURE DOMAIN: Retrieval vs. model → CONTAINMENT: FALLBACK WORKFLOW flagging, or REDUCED AUTONOMY if model-driven → RECOVERY: Restore KB access, or roll back model/prompt change **SYMPTOM: Latency rises** → FIRST DISCRIMINATING SIGNAL: Isolated to one tool/model step, or across the board? → POSSIBLE CAUSES: Specific external API slowdown vs. queue/worker saturation → NEXT CHECK: Per-stage latency breakdown from trace → ISOLATED FAILURE DOMAIN: Specific tool vs. infra → CONTAINMENT: None automatic unless latency causes downstream timeouts → RECOVERY: Provider-side resolution, or scale infra **SYMPTOM: Business outcome declines** → FIRST DISCRIMINATING SIGNAL: Are escalations still being created at normal rate, but Jira resolution rate is falling? → POSSIBLE CAUSES: Brief quality decline (engineering can't act on them), wrong routing, or genuine engineering-side capacity issue (outside agent's control) → NEXT CHECK: Sample recent briefs and routing decisions against outcomes → ISOLATED FAILURE DOMAIN: Agent quality vs. external (engineering capacity) → CONTAINMENT: REDUCED AUTONOMY on brief generation if quality is the cause → RECOVERY: Prompt/retrieval fix, or flag to engineering leadership if capacity is the actual cause (outside agent's remit) --- # 14. CONTAINMENT & KILL-SWITCH PLAN | CONTROL | TRIGGER | SCOPE | IMPACT | REVERSIBILITY | RECOVERY REQUIREMENT | |---|---|---|---|---|---| | DISABLE JIRA CREATION | Duplicate-issue spike or Jira tool failure | Jira write path only | Cases accumulate in pending-escalation state | Fully reversible | Fix dedup/tool issue, verify with test case before re-enabling | | DISABLE WRITE ACTIONS (all) | Cross-task contamination or state-store failure | All write paths (Jira, Slack, Zendesk updates) | Agent becomes read-only/advisory | Fully reversible | Root-cause fix + audit of recent writes | | FORCE HUMAN APPROVAL (all actions) | Confirmed or suspected approval bypass (INV-3) | All consequential actions | Slower throughput, but safe | Fully reversible | Fix bypass logic, verify approval gate in trace before relaxing | | PAUSE WORKFLOW (specific case) | Case-level loop/retry-storm detection | Single case | That case stalls until manually reviewed | Fully reversible | Manual review, then resume or reassign | | STOP NEW TASKS (intake pause) | Queue backlog or upstream instability | New case intake only | In-flight cases continue; no new ones start | Fully reversible | Confirm backlog/instability resolved | | ALLOW IN-FLIGHT COMPLETION | Any partial pause scenario | Cases already past point of no unsafe action | Lets safe work finish while blocking new risk | N/A (a policy, not itself reversible) | Define per-scenario what counts as "safe to finish" | | QUARANTINE FAILED TASKS | Cases hitting INV violations or repeated dead ends | Individual cases | Removed from active processing, held for human review | Reversible after review | Root-cause fix + manual case disposition | | ROLL BACK VERSION | Regression correlated with a specific deploy (Stage 10) | Model/prompt/workflow version | Reverts to prior known-good behavior | Reversible if prior version preserved | Confirm regression resolves post-rollback | | FULL PAUSE (kill switch) | State-store failure, confirmed INV-3/INV-6 violation, or multi-tool cascading failure | Entire agent | No processing at all until manual restart | Reversible | Full incident review + explicit human resume decision | Prefer the narrowest control (single-tool or single-case) before escalating to full pause, consistent with the instruction to contain narrowly when safe — full pause is reserved for source-of-truth failures or safety-invariant violations. --- # 15. RECOVERY & REPLAY PLAN **WHAT CAN RETRY:** Read-only calls (Zendesk fetch, KB search) — freely, with backoff. **WHAT MUST NOT RETRY:** Jira issue creation without an idempotency/dedup check first; any action already confirmed successful. **WHAT CAN REPLAY:** Evaluation/brief-drafting steps, from the last verified checkpoint — safe because they produce no external side effect until the action step. **WHAT REQUIRES IDEMPOTENCY:** Jira issue creation (keyed by Zendesk ticket ID), Slack approval requests (avoid duplicate approval threads for the same decision point). **WHAT REQUIRES RECONCILIATION:** Task state vs. Zendesk open-ticket set (INV-1); task state vs. Jira issue status (INV-9). **WHAT REQUIRES HUMAN REVIEW:** Any case flagged for INV-3, INV-6 violation, or repeated dead-end (Stage 7); any case where automated recovery fails twice. - **RECOVERY INPUT:** Last durable task-state checkpoint for the case, including state version. - **CHECKPOINT:** Written after each verified state transition (not after every tool call) — checkpoint granularity itself is a design choice; UNKNOWN what current granularity is. - **REPLAY BOUNDARY:** Never replay past a confirmed external action (e.g., don't re-run the whole case from scratch if a Jira issue already exists — resume from post-creation step). - **DUPLICATE-SIDE-EFFECT PROTECTION:** Idempotency key check before every external write, independent of whether this is a fresh run or a replay. - **COMPLETION VERIFICATION:** Before marking a case complete/escalated, fetch back the Jira issue and confirm it exists, is linked to the correct ticket, and matches expected fields. --- # 16. HUMAN OPERATIONS MODEL | ROLE | DECISION | EVIDENCE NEEDED | CONTROL AVAILABLE | ESCALATION PATH | |---|---|---|---|---| | On-call Ops | Is a case stuck/lost and does it need manual re-enqueue? | Reconciliation diff, case trace, last state transition | Re-enqueue case, reassign approval | Eng on-call if root cause is systemic | | On-call Engineering | Is a tool/infra failure causing cascading issues? | Tool health signals, error logs, recent deploys | Circuit-break tool, roll back version, trigger full pause | Eng lead for CRITICAL CONTAINMENT decisions | | Approver (consequential actions) | Should this specific action proceed? | Escalation brief, evidence references, case history | Approve/reject in Slack | Reassign to backup approver on deadlock | | Eng Lead / Product | Is the agent's behavior degrading in quality, and should autonomy be reduced? | Quality drift signals (Stage 9), business-outcome trend | Force REDUCED AUTONOMY or PAUSED mode | Leadership if outcome decline is sustained | Every dashboard element below is tied to one of these decisions — no metric is shown without an attached action. --- # 17. DASHBOARD ARCHITECTURE **Dashboard: Are tasks completing?** - QUESTION: Is throughput normal? - METRICS: Completion rate over time, queue depth, cases-in-progress count - TRACE LINK: Drill into any case's full trace - DECISION: Trigger Stage 13 "completion rate drops" diagnostic tree **Dashboard: Where are tasks stuck?** - QUESTION: Which cases haven't advanced, and where? - METRICS: Case age since last transition, breakdown by pipeline stage - TRACE LINK: Per-case trace - DECISION: Ops re-enqueue or manual review **Dashboard: Are tools healthy?** - QUESTION: Is any dependency degrading? - METRICS: Per-tool success rate, latency, retry count - TRACE LINK: Per-tool call log - DECISION: Circuit-break decision (Stage 14) **Dashboard: Are actions safe?** - QUESTION: Any invariant violations on actions? - METRICS: Duplicate-issue count, action-without-approval count - TRACE LINK: Full run trace for flagged cases - DECISION: CRITICAL CONTAINMENT trigger **Dashboard: Are human gates working?** - QUESTION: Are approvals timely and legitimate? - METRICS: Approval wait-time distribution, deadlock count, bypass-check results - TRACE LINK: Approval log per case - DECISION: Reassign approver / force HUMAN-APPROVAL-ONLY **Dashboard: Is behavior degrading?** - QUESTION: Is quality drifting from baseline? - METRICS: Escalation-rate trend, grounding-flag rate, human-override rate - TRACE LINK: Sampled brief content per flagged case - DECISION: REDUCED AUTONOMY trigger **Dashboard: Did a recent change cause regression?** - QUESTION: Does a metric shift align with a deploy? - METRICS: Change log overlay on any of the above trend lines - TRACE LINK: Deploy/version log - DECISION: Rollback candidate (Stage 14) **Dashboard: Are business outcomes holding?** - QUESTION: Do escalations actually get resolved? - METRICS: Jira resolution rate for agent-created issues, time-to-resolution - TRACE LINK: Case-to-Jira-outcome linkage - DECISION: Business-outcome investigation (Stage 9) --- # 18. SLI / SLO PLAN | DIMENSION | SLI | TARGET | |---|---|---| | COMPLETION | % of cases reaching verified completion within expected window | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | CORRECTNESS | % of escalations judged appropriate by engineering (not false positive/negative) | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | TOOL RELIABILITY | Per-tool success rate | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | STATE CONTINUITY | % of multi-day cases resuming without state loss | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | APPROVAL | Median/95th-percentile approval wait time | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | RECOVERY | % of failed cases successfully auto-recovered vs. requiring human intervention | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | | OUTCOME | % of escalated issues reaching engineering resolution | TARGET NOT PROVIDED — ESTABLISH BASELINE FIRST | No numeric SLO is fabricated; each SLI is defined and measurable, awaiting operational baseline data. --- # 19. PRODUCTION VALIDATION DRILLS | SCENARIO | EXPECTED DETECTION | EXPECTED CONTAINMENT | EXPECTED RECOVERY | PASS CONDITION | STATUS | |---|---|---|---|---|---| | Jira outage during issue creation | Tool failure signal, verification-fetch failure | DISABLE JIRA CREATION, hold case in pending state | Resume creation with idempotency check once Jira recovers | No duplicate issue created; case eventually escalates correctly | DESIGNED — NOT EXECUTED | | Zendesk queue backlog spike | Queue depth alert | STOP NEW TASKS (intake pause) | Backlog drains, intake resumes | No case lost; oldest cases prioritized | DESIGNED — NOT EXECUTED | | Model/prompt version change causing escalation-rate spike | Escalation-rate deviation alert | REDUCED AUTONOMY | Rollback or prompt fix, re-baseline | Escalation rate returns to baseline | DESIGNED — NOT EXECUTED | | KB retrieval failure | Retrieval failure signal | FALLBACK WORKFLOW (evidence-gap flag) | KB access restored, flagged briefs re-verified | No unsupported claims reach engineering | DESIGNED — NOT EXECUTED | | Stale state after restart | State-version mismatch on resume | Freeze affected case pending manual check | Correct checkpoint identified, case resumes accurately | Case resumes without contaminated or outdated data | DESIGNED — NOT EXECUTED | | Approval delay beyond SLA | Approval-wait alert | Auto-reminder, then reassignment | Backup approver responds | Case unblocked within revised SLA | DESIGNED — NOT EXECUTED | | Duplicate ticket event (webhook fires twice) | Dedup-check log | Second event rejected before any write | N/A — no duplicate action occurs | Only one Jira issue exists for the ticket | DESIGNED — NOT EXECUTED | | Partial action (Jira created, Slack notify fails) | Post-action verification mismatch | Hold case at "action partially complete," retry notify only | Notify succeeds on retry without recreating Jira issue | Case reaches fully-consistent state | DESIGNED — NOT EXECUTED | | Worker crash mid-case | Resume-from-checkpoint on restart | N/A (automatic) | Resume from last verified checkpoint | No duplicate action, no lost progress | DESIGNED — NOT EXECUTED | | Full recovery/replay after multi-tool outage | Multi-tool failure signal | FULL PAUSE | Staged resume: state store first, then tools, then intake | All in-flight cases reconciled with no duplicates or losses | DESIGNED — NOT EXECUTED | | Kill switch activation and resume | Manual trigger | FULL PAUSE | Manual resume after root-cause review | Agent resumes only after explicit human sign-off | DESIGNED — NOT EXECUTED | --- # 20. OPERATIONAL READINESS GATE **READY WITH OBSERVABILITY GAPS** Rationale: The architecture and invariants are well-defined from the stated workflow and risks, and a complete observability/containment design can be specified. However, no baselines, thresholds, retry limits, SLAs, or historical failure rates were supplied (all marked BASELINE REQUIRED / UNKNOWN), and several topology details (trigger mechanism, orchestration framework, queue infrastructure, secondary Slack fallback) are UNKNOWN. This design is not yet backed by production evidence. --- # AGENT PRODUCTION OPERATIONS PACK ## AGENT OPERATING BOUNDARY The agent monitors unresolved Zendesk tickets, retrieves ticket history and KB documentation, decides whether engineering escalation is warranted, drafts evidence-backed briefs, creates Jira issues (with human approval for consequential actions), routes to engineering teams via Slack, and tracks follow-up through durable task state until resolution is verified. ## MUST-REMAIN-TRUE INVARIANTS No task silently disappears (INV-1); no duplicate Jira issue per ticket (INV-2); no consequential action without approval (INV-3); no stale evidence used past its validity boundary (INV-4); no unbounded tool retry (INV-5); no cross-task state contamination (INV-6); no success reported after failed required action (INV-7); no unbounded reasoning loop (INV-8); no completion with follow-up unresolved (INV-9). ## GOLDEN SIGNALS Reconciliation gap (Zendesk vs. task state), duplicate-Jira-issue count, action-without-approval count, per-tool error rate, case age since last state transition, approval wait duration. ## PRIMARY TRACE Run ID → Task/Case ID → trigger → state version → model decision(s) → context/source references with timestamps → tool calls/results → action → approval record → verification → completion/error → recovery path. ## FIRST DEGRADATION SIGNAL Rising case age without state advancement (stuck-task signal) — it is the earliest cross-cutting indicator regardless of which underlying dependency is at fault. ## FIRST DIAGNOSTIC CHECK Pull the trace for the stuck case(s) and inspect the last recorded tool call and its result — this discriminates tool failure, retrieval failure, approval deadlock, or model loop faster than any aggregate metric. ## STUCK-TASK RULE A case with no state transition beyond the expected processing cadence (BASELINE REQUIRED) triggers a WARNING, escalating to ACTION REQUIRED if sustained past a second threshold. ## HUMAN-GATE HEALTH Tracked via approval wait-time distribution and a join between the action log and approval log; any consequential action lacking a matching approval record is a CRITICAL CONTAINMENT event, not a warning. ## PAUSE CONDITION Durable state-store failure, a confirmed action-without-approval event (INV-3), confirmed cross-task contamination (INV-6), or sustained multi-tool failure cascading across the pipeline. ## DEGRADATION MODE READ-ONLY (retrieval and drafting continue, all writes blocked) is the default safe mode when Jira/Slack write paths fail but read paths remain healthy; HUMAN-APPROVAL-ONLY when approval-bypass risk is suspected but the system is otherwise functional. ## KILL SWITCH Full pause of the entire agent (no new intake, no actions) scoped to state-store failure or confirmed safety-invariant violation; narrower controls (disable Jira creation, disable writes, force approval) are preferred first when the failure is isolated to one path. ## RECOVERY PATH Resume from the last verified task-state checkpoint; never replay past a confirmed external action; re-verify Jira/Zendesk linkage before marking any case complete; reconcile task state against Zendesk and Jira as a standing background job. ## FIRST DASHBOARD "Where are tasks stuck?" — case age since last transition, broken down by pipeline stage, with drill-down to per-case trace. ## FIRST ALERT Action-without-approval-record detected — this is the highest-severity, lowest-tolerance invariant violation (INV-3) and should trigger CRITICAL CONTAINMENT immediately. ## FIRST RELIABILITY DRILL Jira outage during issue creation — it directly tests duplicate-prevention (INV-2), idempotency, and the pending-state fallback, which are the most explicitly stated production risks. ## OBSERVABILITY GAPS No supplied baselines for latency, error rates, approval SLA, or case-processing cadence; unknown trigger mechanism (poll vs. webhook); unknown orchestration framework; unknown queue infrastructure; unknown Slack fallback channel; unknown current checkpoint granularity in task state. ## NEXT OPERATIONAL ACTION Instrument the primary trace (Stage 5) and the reconciliation job (Zendesk open tickets vs. task-state cases) first — this single job directly detects the two most severe stated risks (silent case loss and duplicate Jira issues) and produces the baseline data needed for every threshold marked BASELINE REQUIRED above. ## STATUS READY WITH OBSERVABILITY GAPS ## EVIDENCE BASIS DESIGN ONLY