1. Executive Summary The solution is a durable, event-driven escalation workflow in which a bounded LLM performs investigation and drafting, while deterministic code owns state, permissions, routing validation, idempotency, retries, and approvals. It is a single agent with typed tools, not a multi-agent system. The reason is that every supplied requirement (durable state, async operation, duplicate handling, audit, least privilege, human approval) is a control-plane problem that code solves reliably, whereas the model is needed only for correlation, root-cause hypotheses, and brief writing. The agent monitors unresolved tickets, gathers history and knowledge-base evidence, decides whether escalation is justified, drafts an evidence-cited brief, routes by product area, creates a Jira issue, notifies Slack, and tracks resolution. Refunds, account changes, and customer-facing commitments are never executable by the agent. They are structurally impossible through its credentials and tool surface, and any such need becomes an approval request to a human. 2. Requirements and Boundaries 2.1 Confirmed requirements From the supplied brief: monitor unresolved tickets; retrieve ticket history and documentation; correlate with known issues; judge whether escalation is justified; produce concise evidence-backed briefs; route by product area; track responses and resolution; avoid unauthorized customer commitments; require human approval for refunds, account changes, and other high-impact actions. Integrations are Zendesk, Slack, Jira, and an internal knowledge base. Nonfunctional needs are durable task state, asynchronous workflows, customer-data protection, complete audit trails, duplicate-event handling, bounded retries, and least privilege. The existing Python service exposes ticket retrieval, knowledge-base search, Jira issue creation, Slack notification, and ticket update APIs. PostgreSQL is available, and OAuth2 service credentials are used. Existing interfaces are preserved unless change is justified. 2.2 Assumptions (labeled, revisable) A1: The existing Python service is the only path to the four external systems, so the agent calls it and never calls vendors directly. A2: Ticket events arrive by webhook or polling; the design supports both and treats them identically after normalization. A3: A product-area-to-Jira-project-and-Slack-channel mapping exists or will be owned by humans as versioned configuration. A4: The ticket-update API can write internal notes separately from public replies. If it cannot, internal notes must be the only permitted write mode, enforced in a wrapper. A5: The Jira API supports linking or searching by an external key. If not, the agent's own database is the authority for deduplication. A6: Jira status changes and comments are observable by polling or webhook. This is unconfirmed and drives the tracking design. A7: OAuth2 scopes can be issued per role. If the existing service uses one broad credential, that is a gap to fix first. 2.3 Ambiguities and contradictions to resolve The escalation criteria are not defined (severity, age, customer tier, repeat count), so they are configuration owned by support leadership, not model judgment. It is unclear whether "tracks resolution" includes updating the customer-facing ticket; this design limits it to internal notes and Slack unless a human approves otherwise. The definition of "unresolved" (Zendesk statuses) needs confirmation. The interface change flagged by the preservation rule is limited to adding an idempotency key parameter to Jira creation and a read-only Jira status lookup if none exists. 2.4 Responsibilities and autonomy The model may observe ticket content, search documentation, propose a classification, hypothesize root causes with cited evidence, and draft briefs. Deterministic code may execute the allowed actions: create one Jira issue per approved policy path, post Slack notifications, add internal notes, and update its own state. Humans approve refunds, account changes, any customer-facing statement, low-confidence or ambiguous routing, and any action outside the allowlist. Autonomy level: autonomous for read, draft, and internal escalation; human-gated for everything else. During an initial rollout period, Jira creation itself should also be human-approved (shadow then gated mode; see roadmap). 3. Architecture and Execution 3.1 Components Intake normalizes Zendesk events into a single internal event with a deterministic dedup key (ticket id plus event type plus source event id or updated timestamp). Events are inserted into PostgreSQL with a unique constraint, so duplicates are dropped at the door. The Orchestrator is a state machine, not a free-running loop. Workers claim tasks from PostgreSQL using row locking with skip-locked semantics and a lease, which gives async processing, crash recovery, and per-ticket serialization without new infrastructure. A broker is deliberately not added; Postgres queueing is sufficient at support-ticket volumes, and it is the smallest design that satisfies the durability requirement. Revisit only if throughput measurements demand it. The Investigator is the LLM step. It receives a bounded, sanitized context package and may call only read tools through a tool gateway. The Policy Engine is deterministic code that evaluates the model's structured proposal against configured escalation criteria and hard rules. The Router maps a validated product area to a Jira project, component, and Slack channel from configuration. The Executor performs writes only after the Policy Engine and, where required, the Approval Gate permit them. The Tracker polls or receives Jira updates and advances state. The Approval Gate posts requests to Slack and records human decisions with identity and timestamp. 3.2 Tool gateway All tool calls, from model or code, pass through one gateway that enforces the role's allowlist, validates arguments against schemas, applies timeouts and rate limits, redacts outputs, records a tool-call record, and returns typed errors. The model sees read tools only: get ticket, get ticket history, search knowledge base, search known issues, and lookup Jira status. The model has no write tools. Its output is a structured proposal that code then acts on. This removes tool abuse and prompt-injection-driven writes as a class, since injected text cannot reach a capability the model does not have. 3.3 Lifecycle Receive: a normalized event creates or wakes a task keyed by ticket id. Duplicates and events for closed tasks are ignored and logged. Gather: deterministic code fetches ticket, history, and top knowledge-base and known-issue results, then builds the context package with content wrapped and labeled as untrusted data, truncated to a token budget, with PII minimized to what investigation needs. Investigate: the model returns a schema-validated proposal containing an escalation recommendation, confidence, product area, root-cause hypotheses, and evidence items. Each evidence item must reference a specific ticket message or document id. Verify: code checks that every cited id exists in the retrieved context and that quoted snippets appear in the source. Uncited claims are stripped or the proposal is rejected. Hypotheses are labeled hypotheses, never facts. Decide: the Policy Engine combines the recommendation with configured thresholds. Outcomes are close with note, continue monitoring, escalate, needs human review, or needs approval for a high-impact request detected in the ticket. Route: the product area must map to a known team. Unknown or low-confidence area goes to a human triage queue in Slack, never a guess. Execute: create the Jira issue with an idempotency key, post Slack, add an internal ticket note linking the Jira key. Each side effect is its own recorded step. Track: the Tracker updates state on Jira transitions and comments, alerts on missed response windows, and posts an internal note on resolution. Customer communication about resolution is drafted for a human, not sent. Terminate: completion when Jira is resolved and the ticket is handled by a human, safe termination on retry exhaustion, cancellation, or policy block, each with a recorded reason. 3.4 Decision boundary The model supplies judgment inputs. Code supplies decisions. Escalation occurs only if configured criteria are met and evidence verification passes, so model confidence alone never triggers an action. Model confidence is treated as uncalibrated until measured against labeled outcomes. 4. State, Reliability, and Security 4.1 State and memory Durable state lives in PostgreSQL: events, tasks with an explicit status enum and version column, evidence snapshots, proposals, approvals, external object links (ticket, Jira key, Slack message ts), tool-call records, and an append-only audit log. Transient state is the in-flight context package, rebuilt from durable sources on every attempt, never persisted as raw customer text beyond the evidence snapshot needed for audit. Conversational context is per-task and discarded at termination. Long-term memory is intentionally limited to a reviewed known-issues index and routing configuration curated by humans. The agent does not autonomously write its own long-term memory, which prevents poisoning and drift. Stale state is handled with optimistic concurrency: each transition checks the task version, and a task is re-gathered if the ticket's updated timestamp has advanced since the proposal. Retention: evidence snapshots and audit records follow a configured retention period, with redaction of PII on expiry while preserving audit metadata. 4.2 Reliability Every external call has a timeout, and retries are bounded (assumed default three attempts with exponential backoff and jitter, configurable) and applied only to transient errors and idempotent operations. Authentication and validation errors are not retried. A circuit breaker per integration pauses calls after repeated failure and raises an alert. Idempotency for writes uses the task id plus step name as the key. Before creating a Jira issue, the Executor checks the database for an existing link, and if the record is missing after an ambiguous failure, it searches Jira by the external key (per A5) before retrying, which prevents duplicate issues after timeouts. Multi-step execution is a saga: each step is recorded, a crashed worker resumes from the last completed step, and partial completion (Jira created, Slack failed) retries only the missing step. Compensation is limited to internal notes and Slack correction posts; Jira issues are annotated, not deleted. Model failures: malformed or schema-invalid output gets one repair attempt, then falls to human review. Timeouts and provider outages leave the task queued with backoff, and the ticket remains visible to humans, so failure of the agent never silences an escalation. Conflicting information (ticket says one product, documentation another) lowers confidence and routes to human triage. A dead-letter status with an alert captures tasks exceeding retry bounds. 4.3 Security Least privilege is enforced by credential separation. The read role holds scopes for ticket read and knowledge-base search only. The write role can create Jira issues, post to specified Slack channels, and add internal ticket notes only. No credential held by the agent can issue refunds or change accounts, so approval for those is a technical fact, not a prompt instruction. Secrets come from a secrets manager, tokens are short-lived, and the model never sees credentials. Prompt-injection resistance rests on architecture: untrusted content is delimited and labeled as data, the model has no write tools, outputs are schema-validated, and routing and actions are validated against allowlists. Briefs are treated as untrusted output when rendered, with links and mentions stripped or allowlisted, so an injected ticket cannot cause broad Slack pings or exfiltrate data through URLs. Sensitive data: minimize fields sent to the model, redact obvious secrets and payment data patterns before model calls and before logging, and confirm the model provider's data-handling terms satisfy policy (an unresolved dependency). Slack briefs go only to team channels with membership appropriate to the data, and full customer details stay in Zendesk with a link. Approvals: a high-impact action request creates an approval record with the proposed action, evidence, and requester context. Only authenticated humans in an authorized group can approve, and the approval is bound to the exact action payload hash, expires, and is single-use. The agent never states or implies commitments to customers; drafts for customers are marked draft and require human send. Audit logs are append-only, record actor (agent, human, system), inputs, decisions, and outputs by reference, and are access-controlled. 5. Implementation Blueprint 5.1 Modules Suggested package layout within the existing service: intake (event normalization and dedup), tasks (state machine, queue claim, leases), gateway (tool registry, schemas, role allowlist, redaction, records), agent (context builder, prompt templates, model client, proposal schema, evidence verifier), policy (escalation criteria, hard rules, high-impact detector), routing (versioned mapping), executor (saga steps, idempotency), tracker (Jira sync, SLA timers), approvals (request, decision, expiry), audit (append-only writer), and config. Existing endpoints are wrapped by gateway adapters and left unchanged, except the two additions noted in 2.3. 5.2 Core schemas (PostgreSQL) Table task: id, ticket_id unique, status, version, attempt_count, lease_owner, lease_expires_at, policy_version, prompt_version, created_at, updated_at, terminal_reason. Table event: id, dedup_key unique, ticket_id, type, payload_ref, received_at. Table proposal: id, task_id, model_id, prompt_version, output_json, verification_result, created_at. Table evidence: id, task_id, source_type, source_id, snippet_hash, snapshot_redacted. Table action: id, task_id, step, idempotency_key unique, status, external_ref, attempts, last_error. Table approval: id, task_id, action_type, payload_hash, status, requested_at, decided_by, decided_at, expires_at. Table audit_log: id, ts, task_id, actor, event_type, detail_ref (append-only, insert-only database permissions). Table tool_call: id, task_id, tool, args_hash, status, latency_ms, error_class. 5.3 Task states received, gathering, investigating, verifying, decided, awaiting_approval, executing, escalated, tracking, resolved, closed_no_action, needs_human, dead_letter. Transitions are enforced in one function with version checks; illegal transitions raise and are audited. 5.4 Proposal contract The model returns JSON validated by a strict schema: recommendation (one of escalate, monitor, no_escalation, needs_human), confidence (0 to 1), product_area (must be from the configured enum or unknown), summary (bounded length), hypotheses (each with statement, likelihood label, and evidence ids), evidence (source type, id, quoted snippet, relevance), missing_information, and customer_impact. Validation failure is an error, never silently coerced. 5.5 Brief format The brief is generated from validated fields by a deterministic template, not free model text, containing: issue summary, affected product area, customer impact, timeline, reproduction information if present, evidence citations with links, related known issues, ranked hypotheses marked as unverified, missing information, and what is being requested of engineering. The model contributes bounded text fields only. 5.6 Configuration Versioned configuration holds escalation criteria, product-area routing, confidence thresholds, retry and timeout values, token budgets, redaction patterns, SLA windows, approval-eligible groups, and the action allowlist. Config changes are reviewed, versioned, and recorded on each task by policy_version so decisions are reproducible. 5.7 Dependencies PostgreSQL driver with a migration tool, a schema validation library, an HTTP client with timeout controls, the model provider SDK behind a thin interface (so models can be swapped and mocked), and an OpenTelemetry-compatible tracing library. No new infrastructure beyond what was supplied, other than a secrets manager and monitoring stack if not already present. 6. Testing, Evaluation, and Operations 6.1 Testing Unit: state machine transitions and illegal transitions, dedup key generation, policy rules, redaction, schema validation, evidence verifier (fabricated citation, altered quote), approval hash binding and expiry. Tool and integration: gateway allowlist enforcement per role, timeout and retry behavior, malformed and partial API responses, rate limit responses, idempotency across simulated timeouts, using recorded or fake versions of the four systems. Workflow: end-to-end with fakes covering normal escalation, no escalation, unknown product area, conflicting evidence, duplicate events, out-of-order events, worker crash at each saga step with resume, Slack failure after Jira success, Jira unavailable, model outage, ticket updated mid-investigation, concurrent workers on one ticket. Security and adversarial: tickets containing instructions to refund, ignore rules, reveal prompts, mass-mention, or embed exfiltration links; injected text in knowledge-base articles; attempts to call disallowed tools; expired or replayed approvals; a check that write credentials lack refund and account scopes. Regression: a versioned golden set of real, anonymized tickets with labeled expected recommendation, area, and key evidence, run on every prompt, model, or policy change. 6.2 Evaluation criteria Track escalation precision and recall against human labels, routing accuracy, citation validity (target 100 percent verified), hallucinated-evidence rate (target zero passing verification), schema-valid rate, human override rate, duplicate-issue rate (target zero), time from ticket qualification to escalation, tool-call correctness, latency and cost per task, and calibration of confidence. Targets other than those marked are to be set after baseline measurement, since none were supplied. High-impact action requests must never execute without approval, verified by test and by audit query. 6.3 Observability Structured logs with task id, ticket id, and correlation id on every line, PII redacted. Traces span intake through tracking with spans per tool call and model call. Metrics: queue depth and age, task counts by state, retries, dead letters, integration error rates and latency, circuit breaker state, model latency and token use, approval wait time, and SLA breaches for engineering response. Alerts: dead letters, stuck tasks (lease expiry without progress), integration outage, rising schema failures, spike in escalations (possible injection or incident), and any write attempt denied by the gateway. Operational runbooks cover replay of a task, manual takeover, and disabling the agent. 6.4 Deployment and maintenance Separate development, staging, and production environments with separate credentials and Jira and Slack targets. Prompts, policy, and routing are versioned and released like code, with canary or shadow evaluation before promotion. A global kill switch and per-product-area disable flags allow immediate stop with tasks safely preserved. Rollback is by reverting the configuration or prompt version, and tasks record the version used. Database migrations are backward compatible. Routing tables and the known-issues index have named owners and a review cadence, and the golden set is refreshed periodically. 7. Tradeoffs, Risks, Final Architecture, and Roadmap 7.1 Material tradeoffs Single agent with a deterministic orchestrator versus multi-agent: chosen for auditability and simplicity, at the cost of less flexible reasoning. Postgres-backed queue versus a broker: chosen for fewer moving parts and transactional consistency with state, at the cost of a scaling ceiling that must be monitored. Model without write tools versus model-driven actions: chosen for security, at the cost of more code for proposal handling. Template-generated briefs versus free-form: chosen for verifiability, at some loss of narrative quality. Human review of low-confidence routing adds latency but prevents misrouted engineering work. Redaction reduces model context quality slightly but protects customer data. 7.2 Principal risks and mitigations Incorrect root-cause hypotheses misleading engineers: mitigated by labeling as unverified, mandatory citations, and evidence verification. Escalation noise or misses: mitigated by configured criteria, shadow mode, and measured precision and recall. Prompt injection: mitigated by absent write tools, delimiting, output validation, and scoped credentials. Duplicate Jira issues: idempotency keys, database link check, and Jira search fallback. Data exposure to the model provider or Slack: minimization, redaction, channel controls, and confirmation of provider terms. Broad existing credentials (A7): must be resolved before write enablement. Unknown Jira observability (A6): polling fallback with a longer response-tracking interval. Anti-patterns avoided: letting the model decide permissions, prompt-only safety rules, unbounded retry or context, and agent-authored long-term memory. 7.3 Final architecture One event-driven agent service inside the existing Python codebase. Intake deduplicates events into PostgreSQL. A state-machine orchestrator with leased workers processes one task per ticket. A read-only, schema-validated model step produces a cited proposal that code verifies and a policy engine decides on. A router maps validated areas to teams, an executor runs an idempotent saga to Jira, Slack, and internal notes through a single tool gateway with role-scoped OAuth2 credentials, and a tracker follows Jira to resolution. An approval gate bound to payload hashes controls all high-impact actions, which the agent's credentials cannot perform anyway. An append-only audit log, tracing, and alerts cover operations. 7.4 Roadmap Phase 1, foundations: confirm assumptions A1 to A7, split OAuth2 scopes into read and write roles, add the idempotency key and status lookup to the Jira wrapper, create the schema, state machine, intake dedup, gateway, and audit log. Phase 2, read-only investigation in shadow mode: context builder, model step, evidence verifier, policy engine, and briefs posted only to a private Slack channel; build the golden set and measure precision, recall, routing accuracy, and citation validity. Phase 3, gated execution: executor saga, router, approval gate, with human approval on every Jira creation; run failure and adversarial test suites. Phase 4, tracking and autonomy: tracker, SLA alerts, resolution notes; enable autonomous internal escalation for product areas that meet agreed accuracy thresholds, keeping refunds, account changes, and customer commitments permanently human-gated. Phase 5, hardening: load testing against the Postgres queue, dashboards and runbooks, retention automation, periodic review of routing, known issues, and thresholds. 7.5 Unresolved issues and next steps Unresolved: escalation criteria and thresholds, definition of unresolved statuses, product-area routing ownership, Jira and Slack event capabilities, model provider data terms, retention periods, and success targets. Next steps: hold a requirements review to close these, audit current credential scopes, and approve the schema and state machine as the first implementation deliverable.