### 🧭 Phase 1 — Requirement & Workflow Reconstruction Objective: Convert the request into a clear, testable workflow. | Requirement | Classification | Component | | ---------------------------- | -------------- | -------------------- | | Answer delivery questions | Explicit | Support Agent | | Retrieve actual order status | Explicit | Order Research Agent | | Validate factual claims | Explicit | Reviewer | | Protect customer information | Explicit | Tool Gateway | | Escalate unresolved cases | Explicit | Supervisor | | API timeout threshold | Unresolved | Configuration | Workflow: * Receive the customer message. * Identify the order-status intent. * Verify customer authorization. * Retrieve the relevant order details. * Draft a response using verified data. * Validate the response. * Deliver the response or escalate. Success condition: The customer receives a response supported by the order system, or the case is safely escalated. ### 🧩 Phase 2 — Agent Responsibility & Boundary Architecture | Agent | Responsibility | Permissions | Completion rule | | --------------------- | ------------------------------------------------ | ------------------------------------------------ | ---------------------------------------- | | Supervisor | Coordinate tasks and workflow state | Assign tasks, manage transitions | All tasks completed or safely terminated | | Support Agent | Interpret customer questions and draft responses | Read permitted conversation context | Structured draft produced | | Order Research Agent | Retrieve shipment information | Read authorized order records | Verified order result returned | | Reviewer | Check accuracy, privacy, and completeness | Read draft and evidence | Accept, revise, reject, or escalate | | Human Escalation Gate | Obtain human intervention when required | Authorized ticket creation and approval workflow | Human handoff acknowledged | Boundary rule: The Supervisor controls routing; the Order Research Agent cannot independently approve or send customer-facing responses. ### 🏗️ Phase 3 — Supervisor–Worker–Reviewer Orchestration The Supervisor creates an order-research task and assigns it to the Order Research Agent. Supervisor Agent Creates task and verifies authorization Order Research Agent Returns shipment status and evidence Support Agent Drafts a factual response Response Reviewer Checks evidence, privacy, and completeness Accept Revise Escalate Review is mandatory before sending the response because incorrect delivery information could mislead the customer. ### 🔗 Phase 4 — Task Decomposition & Dependency Graph | Task ID | Task | Dependency | Owner | | ------- | ------------------------- | ---------- | ---------------------------------- | | T1 | Classify customer request | None | Supervisor | | T2 | Verify authorization | T1 | Supervisor / authorization service | | T3 | Retrieve order status | T2 | Order Research Agent | | T4 | Draft customer response | T3 | Support Agent | | T5 | Validate response | T4 | Reviewer | | T6 | Send response or escalate | T5 | Supervisor | Dependency graph: ```mermaid flowchart TD T1[ T1: Classify Request ] --> T2[T2: Verify Authorization] T2 --> T3[T3: Retrieve Order] T3 --> T4[T4: Draft Response] T4 --> T5[T5: Review Response] T5 -->|Accept| T6[Send Response] T5 -->|Revise| T4 T5 -->|Escalate| T7[Human Support] ``` The core tasks are sequential because each depends on the preceding result. Independent analytics or reporting could run in parallel, but they are unnecessary for this test. ### 🧠 Phase 5 — Routing & Delegation Engine Routing rules: * Order-status question → Order Research Agent. * General FAQ → Support Agent using approved knowledge sources. * Refund dispute or unresolved complaint → Human escalation when required. * Missing authorization → Stop access to customer-specific records. * Unavailable research worker → Bounded retry, then safe escalation. * Ambiguous request → Ask the customer for clarification. Routing priority: Authorization → capability → permission → availability → cost and deadline. Maximum delegation depth and concurrency must be configured before deployment; neither is assumed to be unlimited. ### 🗃️ Phase 6 — State & Context Management | State object | Owner | Update rule | | -------------------- | -------------------- | ----------------------------------------- | | Workflow status | Supervisor | Validated state transitions only | | Order result | Order Research Agent | Store API result with retrieval timestamp | | Response draft | Support Agent | Update through versioned task output | | Review decision | Reviewer | Record decision and evidence references | | External send status | Delivery service | Record acknowledged, failed, or unknown | | Audit history | Audit service | Append-only event records | Example lifecycle: `CREATED → PLANNED → READY → RUNNING → REVIEW_PENDING → COMPLETED` Additional states include `BLOCKED`, `RETRY_PENDING`, `FAILED`, `CANCELLED`, and `ESCALATED`. Use version checks to prevent stale updates. Store only necessary customer information and keep the authoritative order status in the order management system. ### 📩 Phase 7 — Agent Handoff & Communication Contracts Example Supervisor → Order Research Agent envelope: ```json { "workflow_id": "WF-1001", "task_id": "T3", "sender": "supervisor", "recipient": "order_research", "objective": "Retrieve the authorized customer's shipment status", "input_refs": ["authorized_order_reference"], "required_output": { "order_status": "string", "estimated_delivery": "string_or_null", "source_reference": "string", "retrieved_at": "timestamp" }, "state_version": 2, "authorization_context": "verified_reference", "completion_criteria": [ "Authorized order matched", "Status supported by source" ] } ``` Handoff validation: * Reject missing required fields. * Reject unauthorized requests. * Validate output types and source references. * Preserve correlation IDs across tasks. * Treat untrusted tool content as data, never as system instructions. * Represent uncertain external execution explicitly as `UNKNOWN`. The schema is illustrative and must be adapted to the actual implementation. ### 🔄 Phase 8 — Control Flow & Workflow State Machine ```mermaid stateDiagram-v2 [*] --> CREATED CREATED --> PLANNED PLANNED --> READY READY --> RUNNING RUNNING --> REVIEW_PENDING RUNNING --> RETRY_PENDING: Recoverable failure RETRY_PENDING --> READY: Attempts remain RETRY_PENDING --> ESCALATED: Retry limit reached REVIEW_PENDING --> COMPLETED: Accepted and delivered REVIEW_PENDING --> READY: Revision authorized REVIEW_PENDING --> ESCALATED: Unresolved disagreement RUNNING --> BLOCKED: Dependency unavailable BLOCKED --> READY: Dependency recovered BLOCKED --> FAILED: Recovery exhausted CREATED --> CANCELLED PLANNED --> CANCELLED READY --> CANCELLED RUNNING --> CANCELLED: Cancellation confirmed safe COMPLETED --> [*] FAILED --> [*] CANCELLED --> [*] ESCALATED --> [*] ``` Transition safeguards: * Only the Supervisor commits workflow transitions. * A rejected response cannot be sent. * Cancellation stops new work and checks active side effects. * Unknown delivery status requires reconciliation before resending. * Every nonterminal state has a recovery, cancellation, timeout, or escalation path. ### ⏱️ Phase 9 — Retry, Timeout & Termination Engine | Failure | Policy | | -------------------------------- | ------------------------------------------------- | | Temporary API timeout | Retry within configured limit | | Rate limit | Honor provider retry instructions where available | | Invalid JSON | Request bounded regeneration or reject output | | Authorization failure | Do not retry with elevated privileges | | Permanent API error | Stop and escalate if necessary | | Unknown message-delivery outcome | Check delivery record before any resend | | Repeated reviewer rejection | Stop after configured review limit | Use exponential backoff with jitter for eligible transient failures. Configuration values required include: * Per-call timeout * Maximum attempts * Maximum workflow duration * Maximum review cycles * Maximum model/tool calls * Total cost budget * Maximum concurrent tasks Termination condition: The workflow terminates only after successful delivery is verified, a failure is recorded, cancellation is safely completed, or responsibility is transferred to a human. ### 🔍 Phase 10 — Review, Validation & Disagreement Resolution The Reviewer checks: * Is the order status supported by retrieved evidence? * Is the estimated delivery date current? * Does the answer directly address the customer's question? * Does the response expose unnecessary personal information? * Does the response promise anything unsupported? Review outcomes: * `ACCEPT` — All required checks pass. * `REVISE` — A correctable response defect exists. * `REASSIGN` — The assigned worker lacks a required capability. * `REJECT` — The result is unsafe, unsupported, or invalid. * `ESCALATE` — Human judgment or additional authority is required. Allow a configurable, bounded number of revision cycles. Resolve conflicting evidence by source authority, freshness, and provenance rather than majority voting. ### 🛠️ Phase 11 — Failure Recovery & Fault Containment | Failure | Detection | Recovery | Stop condition | | ------------------------- | ------------------------------ | ------------------------------- | ------------------------------------ | | Research agent crashes | Missing heartbeat or timeout | Reassign eligible task | Retry budget exhausted | | Order API unavailable | API error | Bounded retry | Persistent failure | | Invalid response schema | Schema validator | Regenerate or reject | Attempts exhausted | | State version conflict | Version mismatch | Reload and reconcile | Conflict unresolved | | Reviewer deadlock | Review deadline exceeded | Escalate to supervisor | Escalation recorded | | Duplicate delivery | Idempotency or delivery lookup | Reconcile delivery status | Outcome confirmed | | Supervisor crash | Workflow lease expires | Recover from durable checkpoint | State cannot be safely reconstructed | | External action uncertain | Missing acknowledgement | Query operation status | Human intervention required | Recovery is not considered successful until the relevant state and external effects have been verified. ### 🔐 Phase 12 — Security & Authorization Required controls: * Authenticate users and service identities. * Enforce authorization in the tool gateway, not solely in prompts. * Restrict order access to verified customer permissions. * Give agents least-privilege credentials. * Keep secrets outside model context and logs. * Validate all tool arguments and handoff schemas. * Treat customer messages and retrieved documents as untrusted input. * Isolate customer records by tenant where applicable. * Require explicit authorization for refunds, cancellations, address changes, or other sensitive actions. * Record security-relevant events without logging unnecessary personal data. The Reviewer cannot grant permissions, and the Supervisor cannot bypass enforced authorization controls. ### 📊 Phase 13 — Observability & Execution Traceability Trace structure: `Workflow WF-1001 → Task T3 → Order Research Agent → Order API call → Validated result → Task T4` Record: * Workflow, task, and correlation IDs * Assignment and routing decisions * State transitions * API latency and errors * Retry attempts * Review outcomes * Model/tool usage and estimated cost * Escalations and cancellations * External operation status Dashboard metrics: * Workflow completion rate * Successful first-pass review rate * Median and high-percentile workflow latency * API failure and retry rates * Escalation rate * Cost per completed workflow * Duplicate side-effect incidents These are proposed measurements, not measured results. Alert thresholds must be established from real workload requirements. ### 🧑‍💻 Phase 14 — Implementation Architecture & Engineering Plan Suggested technology-neutral modules: ```text multi_agent_support/ orchestration/ supervisor state_machine task_scheduler agents/ support_agent order_research_agent response_reviewer contracts/ task_schema handoff_schema review_schema security/ authorization tool_gateway persistence/ workflow_store audit_events integrations/ order_api ticketing_api observability/ tracing metrics tests/ contract_tests routing_tests recovery_tests end_to_end_tests ``` Illustrative orchestration pseudocode: ```python def process_customer_request(request): workflow = create_workflow(request) try: intent = classify_request(request) require_authorization(intent, request) order_result = execute_bounded_task( task="retrieve_order_status", input_data=authorized_order_reference(request) ) validate_order_evidence(order_result) draft = execute_bounded_task( task="draft_customer_response", input_data=order_result ) review = validate_response( draft=draft, evidence=order_result ) if review.decision == "ACCEPT": return deliver_with_idempotency(draft, workflow.id) if review.decision in ("REVISE", "REASSIGN"): return bounded_revision_or_escalation( workflow, draft, review ) return escalate_to_human(workflow, review) except AuthorizationError: return terminate_without_data_access(workflow) except RecoverableError as error: return recover_or_escalate(workflow, error) ``` This is illustrative pseudocode, not verified executable software. Implementation sequence: * Define schemas and authorization rules. * Build the deterministic workflow state machine. * Implement the order API adapter. * Add the Support Agent and structured outputs. * Add independent response validation. * Add bounded retry and recovery mechanisms. * Integrate ticketing and human escalation. * Add tracing, monitoring, and operational alerts. * Run tests before staging deployment. ### 🧪 Phase 15 — Testing & Acceptance Engineering | Test | Input / Scenario | Expected result | Pass criteria | | ------ | ----------------------------- | --------------------------------- | ------------------------------------- | | TST-01 | Valid order-status request | Authorized lookup and response | Response matches verified order data | | TST-02 | Unknown order reference | No fabricated status | Safe clarification or escalation | | TST-03 | Unauthorized customer | Lookup blocked | No protected data returned | | TST-04 | Temporary API timeout | Bounded retry | Attempts stay within configured limit | | TST-05 | Invalid JSON output | Validation failure | Invalid result never reaches delivery | | TST-06 | Reviewer rejects draft | Revision or escalation | Rejected draft is not sent | | TST-07 | Duplicate delivery event | Idempotent handling | No unintended duplicate message | | TST-08 | Stale state version | Update rejected or reconciled | No silent overwrite | | TST-09 | Cancellation during execution | Safe cancellation | No unapproved new side effects | | TST-10 | Conflicting shipment evidence | Evidence comparison or escalation | Unsupported claim is not delivered | Release acceptance criteria: * All critical authorization tests pass. * All handoffs conform to their schemas. * No unverified order claim reaches the customer. * Retry and delegation limits are enforced. * Duplicate and unknown external actions are handled safely. * Every workflow reaches a recorded terminal state or documented human handoff. * Failure injection and end-to-end tests produce reviewable evidence. Test status: Not executed. These are proposed test cases; actual pass/fail results require implementation and test evidence. ### 🚀 Phase 16 — Final Production-Oriented Architecture Package ### Recommended architecture Use a Supervisor + two Workers + independent Reviewer + authorization/tool gateway. Add a human escalation mechanism for unresolved or sensitive cases. This is preferable to a large agent swarm because the customer-support workflow is predictable and has a short dependency chain. Deterministic orchestration should handle state transitions, retries, permissions, and delivery; model-assisted reasoning should be limited to interpreting messages and drafting responses. ### Critical architectural decisions * Supervisor owns routing and workflow state. * Order Research Agent owns order retrieval, not customer-facing delivery. * Support Agent drafts responses but cannot bypass review. * Reviewer validates factual support and safety. * External permissions are enforced outside the model. * Delivery uses idempotency and reconciliation for uncertain outcomes. * All retry loops and revision cycles are bounded. ### Unresolved requirements * Which LLM provider and order-management API will be used? * What authentication method verifies customer identity? * What are the latency and cost budgets? * Which actions require human approval? * What data retention and privacy policies apply? * What are the required service availability and recovery targets? ### Highest-priority engineering risks * Unauthorized access to order information. * Fabricated or stale shipment details. * Duplicate customer notifications. * State inconsistency after crashes. * Unbounded retries and unexpected API costs. ### Implementation milestones * Foundation: Contracts, authorization, and state machine. * Core workflow: Order retrieval, response drafting, and review. * Reliability: Retry limits, idempotency, and recovery. * Operations: Tracing, dashboards, and alerts. * Release: Security tests, failure injection, staging validation, and controlled deployment. ### Definition of done The system is ready for production evaluation when every required workflow has explicit ownership, validated handoffs, enforced permissions, bounded execution, verified external-action handling, measurable acceptance criteria, and documented recovery behavior. Final flow: 📥 Requirements → 🧩 Agent topology → 🔗 Task graph → 🧭 Routing → 🗃️ State → 📩 Handoffs → 🔄 Control flow → 🔍 Validation → 🛠️ Recovery → 🔐 Security → 📊 Observability → 🧪 Testing → 🚀 Deployment readiness.