1. Team and Work Architecture 1.1 Objective and Scope Objective: Investigate complex customer-support issues through specialized agents and produce accurate, evidence-backed resolutions while preserving ownership, authorization, auditability, and controlled autonomy. Required work: Triage the case, gather permitted context, investigate technical symptoms, retrieve authoritative documentation, validate evidence and proposed resolution, escalate unresolved cases, and track the case through closure. Core work: Triage, technical diagnosis, evidence retrieval, resolution validation, case coordination, escalation, and completion tracking. Supporting work: Context normalization, evidence correlation, confidence assessment, audit recording, retry handling, state management, and authorization enforcement. Optional work: Additional diagnostic specialization should be introduced only when workload or domain complexity demonstrates that existing agents cannot handle the responsibility effectively. Unnecessary work: Separate agents for generic routing, generic monitoring, generic summarization, or independent case management are unnecessary because their responsibilities can be contained within the proposed team. Assumption: The ticketing, documentation, application-log, and engineering-case systems expose interfaces suitable for authenticated agent access. This is an implementation assumption, not a supplied capability. 1.2 Recommended Team 1.2.1 Customer Triage Agent Mission: Establish case ownership, understand the customer-reported problem, normalize the initial context, and determine the investigation path. Responsibilities: Interpret the ticket, identify missing information, classify the issue, collect permitted customer and product context, create the investigation task, and assign work to the appropriate specialist. Inputs: Customer ticket, permitted customer context, available product context, and case history. Outputs: Normalized case context, investigation objectives, initial hypotheses, required evidence, and assigned diagnostic tasks. Authority: Read permitted case and customer information and initiate investigation requests. Exclusions: No technical root-cause confirmation, unrestricted customer-data access, refunds, account changes, or engineering escalation approval. Necessity: Required because investigation quality depends on establishing a consistent case context and explicit investigation objective. 1.2.2 Technical Diagnosis Agent Mission: Investigate technical symptoms and determine plausible causes using authorized read-only diagnostic evidence. Responsibilities: Examine application logs and available technical evidence, correlate symptoms, test diagnostic hypotheses, identify evidence gaps, and produce causal findings. Inputs: Normalized case context, diagnostic objectives, application logs, and permitted technical evidence. Outputs: Diagnostic findings, supporting evidence references, confidence assessment, unresolved questions, and recommended next investigative actions. Authority: Perform read-only diagnostic operations within explicitly authorized capabilities. Exclusions: No production modification, customer-account modification, refund, unsupported inference presented as fact, or final customer resolution approval. Necessity: Required because technical investigation is sufficiently specialized and risk-sensitive to warrant separation from customer-facing case handling. 1.2.3 Knowledge Retrieval Agent Mission: Retrieve authoritative product documentation and known-issue information relevant to the investigation. Responsibilities: Search approved documentation, identify applicable procedures and known issues, assess document relevance, detect conflicting or outdated information where possible, and attach traceable evidence. Inputs: Investigation context, diagnostic findings, product identifiers, and search requirements. Outputs: Relevant documentation references, applicable procedures, known-issue evidence, contradictions, and evidence-quality observations. Authority: Read approved knowledge sources only. Exclusions: No customer communication, technical diagnosis ownership, account changes, or independent escalation decisions. Necessity: Required to keep proposed resolutions grounded in approved product knowledge rather than relying solely on diagnostic inference. 1.2.4 Resolution Validation Agent Mission: Determine whether the proposed resolution is sufficiently supported by the available evidence and approved knowledge. Responsibilities: Correlate diagnosis with retrieved evidence, check consistency, identify unsupported claims, verify resolution applicability, assess confidence, and identify remaining uncertainty. Inputs: Diagnostic findings, knowledge evidence, original case context, and proposed resolution. Outputs: Validation result, evidence trace, confidence assessment, identified contradictions, required corrections, and resolution recommendation. Authority: Validate evidence and recommend acceptance, correction, or further investigation. Exclusions: No unilateral customer commitment, refund, account modification, production change, or unauthorized escalation. Necessity: Required as an independent verification boundary between investigation and customer-facing resolution. 1.2.5 Escalation Coordinator Mission: Maintain case continuity when investigation cannot safely or sufficiently resolve the issue. Responsibilities: Determine whether escalation criteria are satisfied, prepare the engineering case using validated evidence, route it through the appropriate authorized mechanism, track its state, and return relevant outcomes to the case workflow. Inputs: Validated investigation package, unresolved evidence gaps, escalation criteria, and case state. Outputs: Escalation request, engineering-case reference, status updates, required follow-up actions, and closure information. Authority: Coordinate authorized engineering escalations; it does not independently exceed established authorization boundaries. Exclusions: No technical diagnosis ownership, unsupported escalation, production modification, refund, or account change. Necessity: Required because unresolved cases need durable ownership and controlled transition to engineering without losing investigative context. 1.3 Responsibility Structure Customer Triage Agent owns case intake and investigation framing. Technical Diagnosis Agent owns technical investigation. Knowledge Retrieval Agent owns authoritative knowledge retrieval. Resolution Validation Agent owns evidence and resolution validation. Escalation Coordinator owns unresolved-case transition and escalation tracking. No agent owns another agent's core responsibility. The validation agent may challenge diagnostic conclusions without becoming the diagnostic owner. The Customer Triage Agent remains the case-context owner while the Escalation Coordinator owns the escalation lifecycle after authorized escalation. 2. Coordination, Workflow, and Control 2.1 Coordination Model Use a supervisor-style orchestration model centered on the case lifecycle rather than unrestricted peer-to-peer delegation. The Customer Triage Agent initiates the investigation and creates explicit work requests for the Technical Diagnosis Agent and Knowledge Retrieval Agent. Diagnosis and knowledge retrieval should execute in parallel when both can proceed independently. The Resolution Validation Agent receives their outputs only after sufficient evidence is available. Validation determines whether the case can proceed to resolution, requires additional investigation, or meets escalation conditions. The Escalation Coordinator receives only validated escalation packages. Direct agent-to-agent communication should be limited to defined task messages and result messages. 2.2 Message Contracts Every investigation task should contain a case identifier, correlation identifier, task identifier, requested responsibility, relevant context, authorization scope, expected output, and contract version. Every result should contain the originating task identifier, findings, evidence references, uncertainty, status, and required next action. Versioned contracts should protect agents from incompatible message changes. Idempotency keys should prevent duplicate diagnostic or escalation execution. Correlation identifiers should connect ticket activity, agent execution, tool calls, evidence, validation, and escalation records. Asynchronous messaging should be used for investigations that can exceed interactive response limits. 2.3 State Architecture Durable case state should contain ownership, lifecycle status, investigation objectives, agent results, evidence references, validation status, escalation status, and completion information. Task state should track individual agent work, retries, timeouts, dependencies, and completion. Agent-local state should contain only temporary execution context that does not need to survive independently. Conversational state should remain limited to information necessary for the current customer interaction. Shared mutable state should be minimized. Agents should exchange explicit results rather than continuously modifying a common working context. State ownership should remain centralized at the case/workflow level to avoid conflicting agent interpretations. Stale results must be detected through case-version or task-version checks before being incorporated into the active case. 2.4 End-to-End Workflow Case initiation: A complex ticket enters the workflow. Assignment: Customer Triage Agent claims ownership and establishes the investigation objective. Context gathering: Permitted customer, product, ticket, and case information is collected. Delegation: Technical Diagnosis Agent receives the diagnostic task while Knowledge Retrieval Agent receives the knowledge-search task. Parallel execution: Both agents investigate independently where dependencies permit. Synchronization: Their results are correlated against the same case and investigation version. Evidence evaluation: Resolution Validation Agent compares diagnosis, documentation, and case context. Correction: If evidence is insufficient or contradictory, validation returns explicit gaps to the appropriate agent rather than restarting the entire workflow. Resolution: A sufficiently supported resolution is prepared for customer handling. Escalation: If the issue remains unresolved or meets defined escalation criteria, the validated evidence package moves to the Escalation Coordinator. Tracking: Escalated work remains associated with the original case until an authorized outcome is received. Completion: The case closes only when the required resolution or escalation outcome is recorded and ownership is explicitly transferred or returned. Timeout: Work exceeding its bounded execution window becomes a recoverable task state rather than an implicit success or failure. Failure: Tool or agent failure triggers bounded recovery; persistent failure moves to controlled escalation. Missing information: The workflow records the missing evidence and requests only information necessary for the next decision. Uncertainty: Uncertain conclusions remain explicitly marked and cannot silently become confirmed customer resolutions. 2.5 Autonomy and Approval Autonomous actions: Case classification, evidence retrieval, read-only diagnosis, evidence correlation, validation, and preparation of escalation material. Controlled actions: Customer-facing resolution acceptance, engineering escalation, refunds, account changes, and other high-impact actions requiring authorization. Human approval should be inserted at the point where an action exceeds the agent's defined authority. Agents should never gain additional authority merely because another agent recommends an action. Conflicting findings should trigger evidence reconciliation or human review rather than majority-based decision-making. 3. Reliability, Security, Scalability, and Risk 3.1 Reliability Controls Agent failure: Mark the task failed or unavailable, retry within defined bounds, and reroute only when the alternative remains within the responsibility boundary. Tool failure: Retry transient failures with bounded backoff and preserve the original task context. Incomplete evidence: Return the case to the responsible investigation stage rather than treating missing evidence as confirmation. Conflicting results: Require validation and reconciliation before resolution. Duplicate execution: Use task identifiers and idempotency keys. Timeout: Persist the task state and resume or escalate according to defined timeout policy. Partial completion: Preserve completed evidence and restart only the incomplete responsibility. Cascading failure: Prevent repeated downstream requests when prerequisite services remain unavailable. Agent unavailability: Preserve case ownership and transition the affected task into recoverable or escalation state. Termination: Stop investigation when resolution is validated, escalation is authorized and transferred, or the case reaches an explicit terminal state. 3.2 Evidence and Accuracy Every material diagnostic conclusion should reference the evidence supporting it. Documentation-derived claims should identify the source used. Validation should distinguish observed evidence, inferred cause, documented behavior, and unresolved uncertainty. Conflicting evidence should prevent automatic resolution acceptance until reconciled. Confidence should be evidence-based rather than generated solely from model self-assessment. Verification intensity should increase when evidence is incomplete, consequences are higher, or the proposed action exceeds ordinary read-only investigation. 3.3 Security and Governance Agent identities must be authenticated. Authorization must be capability-based and responsibility-specific. The Technical Diagnosis Agent receives read-only diagnostic access. The Knowledge Retrieval Agent receives access only to approved knowledge sources. Customer-sensitive information should be minimized before being passed between agents. Agents should receive only the fields required for their assigned task. Refunds, account changes, and engineering escalations require the appropriate authorization boundary. Audit records should capture agent identity, case identity, task, tool invocation, evidence references, decision, authorization context, and outcome. Separation of duties prevents the same execution path from independently investigating, validating, and authorizing high-impact actions. Sensitive cases require restricted access throughout the workflow, not merely at initial intake. 3.4 Scalability Diagnosis and knowledge retrieval should scale independently because their workloads can differ. Parallel investigation should be preferred when dependencies do not require serialization. The validation stage is a potential coordination hotspot and should remain focused on evidence synthesis rather than performing unrelated investigation. Excessive context transfer should be avoided by passing targeted evidence and references instead of complete case histories. Agent count should remain fixed unless measurable workload or domain requirements justify specialization. Asynchronous task execution allows long-running investigations without blocking customer-facing workflows. Queue-based execution can absorb workload bursts while preserving durable task state. 3.5 Anti-Patterns Agent proliferation: Avoid creating separate agents for every small activity. Responsibility overlap: Do not allow multiple agents to independently own the same decision. Circular delegation: Agents must not delegate tasks back and forth without explicit ownership. Shared-state contention: Avoid unrestricted multi-agent mutation of the same case state. Communication storms: Use structured task boundaries rather than continuous agent conversation. Uncontrolled autonomy: Prevent agents from converting recommendations into unauthorized actions. Weak validation: Do not allow the diagnosis agent to serve as the sole authority for customer-facing resolution. Single-point orchestration failure: Persist workflow state so orchestration recovery does not destroy active investigations. Uncontrolled retries: Apply bounded retries and idempotency to prevent repeated tool operations. Excessive human intervention: Reserve human approval for consequential or explicitly restricted decisions rather than routine investigation. 4. Final Architecture, Rationale, and Implementation 4.1 Final Recommended Architecture Customer Triage Agent: Owns case interpretation, initial context, investigation framing, and case-context continuity. Technical Diagnosis Agent: Owns read-only technical investigation and causal analysis. Knowledge Retrieval Agent: Owns retrieval and traceability of approved product knowledge. Resolution Validation Agent: Owns independent evidence validation and resolution readiness. Escalation Coordinator: Owns authorized escalation preparation, routing, tracking, and return of engineering outcomes. Workflow orchestration: Coordinates lifecycle transitions, dependencies, retries, state persistence, authorization checks, and completion. Case state store: Maintains durable case and task state while minimizing shared mutable state. Evidence layer: Maintains references connecting conclusions to logs, documentation, ticket context, and validation results. 4.2 Architectural Decisions 4.2.1 Five-Agent Team Problem: Complex support investigation requires multiple specialized responsibilities. Chosen approach: Five specialized agents with explicit ownership. Justification: Each agent represents a distinct decision or expertise boundary required by the supplied workflow. Tradeoff: More coordination exists than in a single-agent design, but specialization provides clearer authority, validation, auditability, and parallel execution. 4.2.2 Supervisor-Orchestrated Coordination Problem: Unrestricted peer-to-peer collaboration increases coupling and makes ownership ambiguous. Chosen approach: Central workflow orchestration with bounded agent interactions. Justification: The workflow has explicit lifecycle stages and dependencies. Tradeoff: The orchestrator becomes an important infrastructure component and must itself be highly available and state-aware. 4.2.3 Parallel Diagnosis and Knowledge Retrieval Problem: Technical investigation and documentation retrieval can often proceed independently. Chosen approach: Execute both activities concurrently after triage. Justification: Reduces unnecessary serialization and shortens investigation time. Tradeoff: Results require synchronization and version checks before validation. 4.2.4 Independent Resolution Validation Problem: Diagnostic reasoning alone may contain unsupported conclusions. Chosen approach: Separate validation responsibility. Justification: Creates an explicit evidence-control boundary before resolution. Tradeoff: Adds one processing stage and therefore some latency. 4.2.5 Durable Case State Problem: Investigations may be asynchronous and can span multiple execution cycles. Chosen approach: Persist case and task state independently of individual agent execution. Justification: Supports recovery, long-running investigations, escalation tracking, and reliable ownership. Tradeoff: Requires state lifecycle management and stale-context controls. 4.2.6 Controlled Escalation Problem: Unresolved cases require engineering involvement without uncontrolled autonomy. Chosen approach: Escalation Coordinator prepares and tracks authorized escalations using validated evidence. Justification: Preserves continuity and prevents unsupported escalation decisions. Tradeoff: Some cases require an additional authorization or human-review step. 4.3 Practical Implementation Orchestration: Implement the case lifecycle as a durable workflow with explicit states, transitions, dependencies, timeout handling, and terminal conditions. Runtime boundaries: Run each agent with only the tools and credentials required for its responsibility. Communication: Use structured, versioned messages with correlation and idempotency identifiers. State: Persist case state and task state separately from ephemeral agent execution context. Tools: Enforce tool permissions at the infrastructure layer rather than relying solely on agent instructions. Observability: Record workflow transitions, agent execution, tool calls, evidence references, validation decisions, failures, retries, and authorization events. Security: Apply authenticated identities, least privilege, restricted sensitive-case access, data minimization, and capability-specific authorization. Validation: Require evidence-backed validation before customer resolution and escalation. Recovery: Use bounded retries for transient failures, persisted task state for recovery, duplicate suppression, and controlled escalation for persistent failures. Testing: Test individual agents, message contracts, authorization boundaries, workflow transitions, stale-context handling, failure recovery, evidence validation, duplicate suppression, and complete case lifecycles. Deployment: Deploy agents independently where operationally useful while maintaining centralized workflow and policy controls. Monitoring: Track task latency, failure rates, retry frequency, unresolved cases, validation failures, escalation rates, stale tasks, and authorization exceptions. Essential mechanisms: Durable workflow state, capability-based authorization, evidence traceability, validation, idempotency, bounded retries, audit logging, and explicit ownership. Conditional mechanisms: Additional specialized agents, sophisticated routing, or more complex coordination should be introduced only when actual workload or domain evidence justifies them. 4.4 Collective System Behavior The Customer Triage Agent establishes what needs to be solved. The Technical Diagnosis Agent determines what the technical evidence indicates. The Knowledge Retrieval Agent establishes what approved knowledge supports. The Resolution Validation Agent determines whether the proposed resolution is adequately supported. The Escalation Coordinator preserves ownership when the team cannot safely complete the investigation. Together, these boundaries provide investigation, evidence retrieval, independent validation, controlled escalation, and lifecycle continuity without introducing separate agents for activities that can remain within existing responsibilities. 4.5 Final Recommendation Team composition: Use five agents with the defined responsibility boundaries. Responsibility coverage: Assign one accountable owner to each major investigation and lifecycle responsibility. Coordination: Use durable supervisor-style orchestration with bounded, structured agent communication. Autonomy: Permit autonomous read-only investigation and evidence gathering while controlling consequential actions through authorization boundaries. Reliability: Combine durable state, bounded retries, idempotency, timeout handling, validation, and controlled escalation. Scalability: Parallelize diagnosis and knowledge retrieval and scale their execution independently. Security: Enforce least privilege, data minimization, authenticated identities, restricted access, auditability, and separation of duties. Governance: Keep customer-impacting and otherwise restricted actions behind explicit authorization. Primary risks: Evidence quality, stale context, coordination failure, excessive shared state, uncontrolled retries, ambiguous ownership, and unauthorized actions. Key implementation considerations: Make workflow state, message contracts, authorization, evidence traceability, validation, and recovery mechanisms first-class architectural components. Final architecture: A five-agent, supervisor-orchestrated team with parallel investigation, independent resolution validation, durable case state, controlled escalation, and capability-limited tools provides the required coverage while keeping agent count, overlap, and coordination complexity bounded.