## Phase 1: System Context Reconstruction 🔍 Objective: Understand the business problem and reconstruct the complete system context. | Component | Analysis | | ----------------- | --------------------------------------------------------------------------------------- | | Primary problem | Manual customer support consumes time and resources. | | Inputs | Customer messages, order IDs, account details, and support history. | | Processing | Classify queries, retrieve verified information, draft responses, and validate actions. | | Outputs | Customer replies, escalation tickets, and audit records. | | Users | Customers, support agents, and administrators. | | Dependencies | Store API, customer database, knowledge base, and email service. | | Success condition | At least 60% of routine queries handled safely, subject to measurement. | Explicit requirements - Protect customer information. - Verify order details before answering order-specific questions. - Enforce refund authorization rules. - Escalate unresolved or exceptional cases. Assumptions - The store API provides authenticated order lookup. - The knowledge base contains current shipping and refund policies. - Support agents can review escalated cases. Proposed decision: Use a hybrid architecture combining deterministic workflow logic with AI-based classification and response generation. Unresolved question: Which support platform and order-management system will be used? Test result:  Pass  The business context, dependencies, assumptions, and unknowns are explicitly separated. ## Phase 2: Workflow and Task Decomposition 🧩 The support workflow can be divided into the following tasks. | Task | Classification | Primary owner | | ---------------------------- | ------------------- | --------------------------- | | Receive and validate message | Deterministic | Workflow service | | Classify customer intent | AI-assisted | Triage Agent | | Retrieve order status | Deterministic/API | Order Lookup Service | | Retrieve relevant policies | Retrieval-based | Knowledge Retrieval Service | | Draft customer response | AI-assisted | Response Agent | | Validate factual claims | AI-assisted + rules | Quality Review Agent | | Approve exceptional refund | Human-controlled | Authorized support manager | | Send approved response | Deterministic/API | Messaging Service | | Escalate unresolved cases | Deterministic | Workflow service | Key architectural finding: Order retrieval, permissions, retries, and message delivery should not require separate AI agents. They can be handled more reliably through conventional services. Test result:  Pass  Tasks have defined ownership and appropriate execution methods. ## Phase 3: Single-Agent vs. Multi-Agent Architecture ⚖️ | Criterion | Single agent | Multi-agent | Hybrid | | ---------------------------- | -------------- | --------------------------------------- | --------------------------------------- | | Initial simplicity | High | Low | Medium | | Specialized responsibilities | Limited | High | High | | Deterministic controls | Possible | Possible | Strong fit | | Operational complexity | Low | High | Medium | | Cost control | Easier | Harder | Easier to optimize | | Best use | Simple FAQ bot | Highly independent specialist workflows | Support automation with safety controls | Selected option: Hybrid architecture. Why? - AI handles ambiguous language and response drafting. - Deterministic services handle order lookup and message delivery. - Independent review checks important claims before responses are sent. - Humans retain control over exceptional refunds and sensitive cases. A multi-agent approach is justified when triage, response generation, and independent quality review provide measurable benefits. If testing shows that a single agent performs equally well at lower cost, simplify the design. Test result:  Pass  The architecture is justified by responsibilities and risk, not by agent count. ## Phase 4: Agent Role Discovery 🤖 The minimum proposed starting design has three AI agents. Agent 1: Customer Support Triage Agent Classification and routing - Mission: Identify customer intent and urgency. - Inputs: Customer message and minimal conversation context. - Outputs: Structured issue category, confidence, and routing recommendation. - Tools: Approved classification and routing interface. - Authority: Recommend routing; cannot issue refunds or send messages. - Non-responsibility: Does not independently resolve orders or authorize financial actions. Agent 2: Support Resolution Agent Research and drafting - Mission: Prepare accurate answers to customer questions. - Inputs: Classified issue, authorized order data, and relevant policies. - Outputs: Draft response with supporting evidence and proposed next steps. - Tools: Read-only order lookup and knowledge retrieval through authorized services. - Authority: Can prepare drafts but cannot independently authorize refunds. - Non-responsibility: Cannot change orders, modify customer records, or bypass policy. Agent 3: Response Quality Review Agent Independent validation - Mission: Check draft responses for factual accuracy, policy compliance, and privacy risks. - Inputs: Draft response, cited evidence, and applicable policy rules. - Outputs: Pass, revise, or escalate decision with reasons. - Tools: Read-only policy and evidence access. - Authority: Can reject a draft; cannot approve exceptional refunds or override access controls. - Non-responsibility: Does not rewrite the entire support workflow or send messages. Shared services, not AI agents: - Workflow orchestrator - Order Lookup Service - Knowledge Retrieval Service - Authorization and policy engine - Messaging Service - Audit and monitoring service Test result:  Pass  Each agent has a distinct mission and defined authority. ## Phase 5: Responsibility and Ownership Matrix 📋 | Task | Primary owner | Supporting role | Final authority | | -------------------------- | --------------------------- | -------------------------- | ---------------------- | | Classify query | Triage Agent | Workflow service | Workflow rules | | Verify order | Order Lookup Service | Resolution Agent | API permissions | | Retrieve policy | Knowledge Retrieval Service | Resolution Agent | Policy source | | Draft answer | Resolution Agent | Triage Agent | Quality checks | | Review answer | Quality Review Agent | Resolution Agent | Review rules | | Send routine reply | Messaging Service | Workflow service | Configured send policy | | Approve exceptional refund | Support manager | AI agents provide evidence | Authorized human | | Record audit trail | Audit service | All workflow components | System policy | Ownership rule: Every task has one primary owner. Supporting agents do not inherit the owner's authority. Test result:  Pass  No critical task has ambiguous ownership. ## Phase 6: Multi-Agent Topology Design 🕸️ Recommended topology: Supervisor-worker with sequential validation and deterministic services. Customer Message Workflow Orchestrator Validation, state, routing, and deadlines Triage Agent Resolution Agent Reads verified order data and policy evidence Quality Review Agent Pass Messaging service sends the permitted response Escalate Human reviews unresolved or high-risk cases Parallel execution should be introduced only when independent tasks, such as retrieving shipping information and checking a policy, can safely run concurrently. Test result:  Pass  The topology includes clear routing, validation, and escalation paths. ## Phase 7: Orchestration and Control Flow 🔄 1. Receive a customer message and assign a unique request ID. 2. Validate the input and apply rate limits. 3. Invoke the Triage Agent. 4. Route the request according to issue type and confidence. 5. Retrieve only the order and policy data required. 6. Invoke the Resolution Agent to draft an evidence-based response. 7. Invoke the Quality Review Agent. 8. If the draft passes, apply deterministic send-policy checks. 9. Send the permitted response and record the outcome. 10. If the draft fails, allow one bounded revision attempt. 11. Escalate persistent failures, low-confidence cases, and exceptional requests. 12. Terminate when the request is resolved, escalated, or safely blocked. Loop prevention rules - Maximum one automatic revision after a failed quality review. - Use request IDs and idempotency keys to prevent duplicate sends. - Set timeouts for all external calls. - Never retry a refund or other financial side effect blindly. - Preserve the request state when execution stops. Test result:  Pass  Retries and termination conditions are explicit. ## Phase 8: Communication and Handoff Contracts 📨 Example: Resolution Agent to Quality Review Agent. Handoff contract - Sender: Resolution Agent - Receiver: Quality Review Agent - Trigger: A response draft is ready. - Context: Customer issue category and minimum required conversation details. - Required fields: `request_id`, `issue_type`, `draft_response`, `evidence_references`, `policy_version`. - Expected output: `PASS`, `REVISE`, or `ESCALATE`, with reasons. - Validation: Required fields must exist; evidence references must be resolvable. - Failure response: Reject malformed output or route to human review. - Correlation: The same request ID follows the workflow. Do not transfer: Unrelated customer records, complete database exports, credentials, or irrelevant conversation history. Test result:  Pass  The handoff contains defined fields, validation, and failure behavior. ## Phase 9: Memory, Context, and State Architecture 🧠 | Data category | Storage approach | Access | | ------------------- | ----------------------------- | -------------------------------- | | Working context | Per-request context | Relevant agents only | | Workflow state | Durable state store | Orchestrator | | Order information | Existing order database/API | Authorized services | | Policies | Versioned knowledge base | Retrieval and review components | | Agent-local context | Temporary task context | Individual agent | | Audit history | Append-only audit records | Authorized administrators | | Long-term knowledge | Approved knowledge repository | Explicitly authorized components | Rules - Store workflow state separately from model-generated text. - Treat order APIs and approved policy documents as authoritative sources. - Keep personal data only as long as operational and legal requirements permit. - Reject stale policy versions where current policy is required. - Resolve conflicting state through deterministic versioning or transaction rules. - Do not let an agent directly rewrite trusted order data or audit history. Test result:  Pass  State ownership and access boundaries are defined. ## Phase 10: Tools, Data, and Capability Architecture 🛠️ | Tool/service | Authorized component | Permission | | ----------------- | -------------------- | -------------------------------------- | | Order API | Order Lookup Service | Read-only, scoped lookup | | Customer database | Authorized backend | Minimum necessary fields | | Knowledge base | Retrieval Service | Read-only | | Email service | Messaging Service | Send only under configured policy | | Policy engine | Orchestrator/backend | Enforce action permissions | | Audit service | Backend components | Append records; restricted read access | Important distinction: - Capability means a component can perform a type of task. - Tool access means it can invoke a particular integration. - Permission defines which operations are allowed. - Authority determines which decisions it may make. An AI agent capable of writing a refund recommendation must not automatically receive permission to issue a refund. Test result:  Pass  Tool access and business authority are separated. ## Phase 11: Security and Human Control 🔐 Low risk: Automated within policy Classifying routine questions, retrieving authorized order status, and sending validated answers about verified facts. Review required Conflicting order information, uncertain delivery dates, policy ambiguity, and repeated quality-review failures. High risk: Explicit authorization Exceptional refunds, account changes with material consequences, disclosure of sensitive data, and irreversible actions. Security controls: - Authenticate customers before revealing private order details. - Enforce permissions in backend services, not solely in prompts. - Treat customer messages as untrusted input. - Prevent prompt injection from granting tool access. - Never expose API keys or credentials to agents. - Require explicit human approval for exceptional refunds. - Log privileged actions and their authorization evidence. Test result:  Pass  High-impact actions have explicit authorization boundaries. ## Phase 12: Failure, Recovery, and Resilience ⚠️ | Failure | Detection | Recovery | | --------------------------- | ----------------------- | ------------------------------------ | | Missing order ID | Input validation | Ask customer for required details | | Order API timeout | Timeout monitoring | Bounded retry, then escalation | | Unsupported customer query | Triage confidence/rules | Human review | | Incorrect factual draft | Evidence validation | One revision, then escalation | | Invalid agent output | Schema validation | Reject and retry within limits | | Messaging failure | Delivery status | Idempotent retry or queue for review | | Unauthorized refund attempt | Permission check | Block and audit | | Stale policy data | Version validation | Refresh source or escalate | Test result:  Pass  Failures are detected, contained, and assigned recovery paths. ## Phase 13: Cost, Latency, and Resource Architecture 💰 The system should measure actual performance before setting financial or latency claims. Optimization plan - Use deterministic routing for common, well-defined query types. - Invoke only the agents required for each case. - Cache safe, non-sensitive policy retrieval results where appropriate. - Parallelize independent read-only requests when beneficial. - Avoid repeatedly passing the full conversation to every agent. - Track model calls, tool calls, retries, token usage, and end-to-end response time. Key metrics - Average cost per resolved query - Median and 95th-percentile response time - Agent invocation count per request - API failure and retry rates - Escalation rate - Incorrect-answer rate Test result:  Pass  Efficiency measures are defined without inventing benchmark figures. ## Phase 14: Testing and Evaluation Architecture 🧪 | Test category | Sample test | Acceptance condition | | ------------------ | -------------------------------- | ----------------------------------- | | Unit test | Classify a shipping question | Correct schema and routing | | Handoff test | Pass draft to reviewer | Required fields are present | | Integration test | Retrieve an order | Only authorized fields are returned | | Security test | Request another customer's order | Access is denied | | Authorization test | Attempt unauthorized refund | Action is blocked | | Failure test | Simulate order API timeout | Bounded recovery occurs | | Loop test | Repeated review failure | Workflow terminates or escalates | | Regression test | Re-run approved test set | No unacceptable degradation | | End-to-end test | Routine delivery-status query | Verified answer reaches customer | | Privacy test | Include unrelated sensitive data | Unnecessary data is not propagated | Initial measurable targets - Routine query automation: at least 60%, measured on a representative test set and production sample. - Unauthorized refund execution: zero in authorization tests. - Duplicate message sends: zero in idempotency tests. - Critical workflow: every failure path must terminate or enter a defined escalation state. - Response correctness: establish a baseline and an approved minimum before production rollout. These are proposed acceptance criteria, not completed test results. Test result:  Pass  Critical requirements map to concrete validation methods. ## Phase 15: Implementation Boundaries and Roadmap 🧱 1. Build the core workflow. Implement message intake, request IDs, state transitions, and termination rules. 2. Define agent contracts. Create strict input/output schemas for triage, resolution, and review. 3. Implement deterministic services. Connect authorized order lookup, policy retrieval, and messaging. 4. Add orchestration. Implement routing, handoffs, timeouts, and bounded retries. 5. Add security controls. Enforce authentication, authorization, privacy, and human approval. 6. Add recovery and audit. Handle external failures and record decisions. 7. Create evaluations. Run unit, integration, security, and end-to-end tests. 8. Pilot and optimize. Test on a limited set of routine support queries, measure quality and cost, and expand gradually. Test result:  Pass  Implementation follows the workflow's dependencies and risk profile. ## Phase 16: Final Multi-Agent System Architecture Blueprint 🏆 ### 1. Final architecture summary | Decision | Recommendation | | --------------------- | ------------------------------------------------------------- | | Recommended AI agents | 3 to start | | Topology | Supervisor-worker with sequential review | | Orchestration | Deterministic workflow engine | | Communication | Structured, schema-validated handoffs | | Memory | Per-request context plus durable workflow state | | Tools | Scoped APIs and dedicated backend services | | Authority | Least privilege with explicit approval gates | | Failure strategy | Bounded retries, safe fallback, human escalation | | Evaluation | Unit, integration, security, regression, and end-to-end tests | | First milestone | Build and test the routine order-status workflow | ### 2. Highest architectural risks - Incorrect or stale order information. - Unauthorized access to customer records. - Incorrect refund-policy interpretation. - Duplicate messages or financial side effects. - Excessive AI calls that increase cost without improving quality. - Poor escalation handling that leaves customers without resolution. ### 3. Final acceptance checklist Architecture readiness checklist 0 of 8 Every important task has one primary owner. Agent inputs and outputs use validated schemas. Order lookup and message delivery are controlled by backend services. High-impact actions require explicit authorization. Retries and revision attempts are bounded. Sensitive data is limited to what each task requires. All critical failure paths terminate or escalate. Evaluation metrics are measured before expanding automation. 8 items remain unchecked. ### 🎯 ARCHITECTURE DECISION Build this multi-agent architecture because customer-support triage, evidence-based response drafting, and independent quality review can provide useful separation of responsibility, while deterministic services enforce order access, permissions, and message delivery. Do not build these unnecessary components because additional agents, unrestricted shared memory, autonomous refund execution, and complex distributed infrastructure would increase cost and operational risk without demonstrated benefits. Overall sample assessment:  Design review passed The proposed architecture covers all 16 phases of the supplied Multi-Agent System Architecture Engineer specification. This is a successful structured design exercise, not a claim that the system has been implemented or that live tests have passed.