AI-Agent Backend Architecture Blueprint: Multi-Hospital Healthcare Network Labels used throughout: [Fact] is supplied by you, [Assumption] is mine and needs validation, [Rec] is a recommendation, [Open] is an unresolved decision. 1. Executive Summary 1.1 The recommended design is a modular monolith for the agent control plane, plus a small number of separately deployed services only where isolation is justified. Those are the EHR/FHIR integration gateway, the PHI-handling retrieval service, and the notification service. [Rec] 1.2 Orchestration uses LangGraph with a supervisor-worker pattern. Each agent is a bounded workflow with a narrow tool set. Postgres-backed checkpoints make runs durable, and human approval gates cover any action with patient-facing or record-changing impact. [Rec] 1.3 Agents handle administrative work only: scheduling, inquiries, discharge logistics, record retrieval, and reminders. They never diagnose, triage clinically, or interpret results. Clinical-sounding requests are detected and routed to a qualified human. [Fact: clinical decisions stay with professionals] 1.4 Tenant isolation is per hospital. Every request carries a tenant identity that is enforced at the API, orchestration, tool, database (row-level security), cache, vector, and storage layers. [Fact: hospital-level isolation] 1.5 The three biggest risks are PHI leakage across tenants or into model prompts, prompt injection leading to unauthorized tool actions, and EHR integration fragility. Each has layered mitigations in sections 9 and 16. 1.6 The biggest unknowns are the model provider and its HIPAA terms, Epic integration specifics, and traffic volumes. Section 2 lists the validations needed. 2. Requirements, Constraints, and Assumptions 2.1 Supplied facts 2.1.1 12 hospitals and 35 clinics. 2.1.2 Agents for scheduling, patient inquiries, discharge coordination, record retrieval, and follow-up reminders. 2.1.3 Stack: Python, FastAPI, LangGraph, AWS, Kubernetes, PostgreSQL, Redis, S3, and a vector database. 2.1.4 Integrations: Epic EHR, FHIR APIs, Microsoft Teams, Okta, hospital scheduling systems, SMS services. 2.1.5 HIPAA, strict privacy, RBAC, encrypted storage, audit trails, high availability, hospital-level tenant isolation, reliable integrations, and scalable processing. 2.2 Functional requirements derived from the facts 2.2.1 Agents must read and write scheduling data through authorized channels. 2.2.2 Record retrieval must honor the requesting user’s role and relationship to the patient. 2.2.3 Discharge coordination must produce tasks, notifications, and handoffs, not clinical instructions. 2.2.4 Reminders must be reliable, deduplicated, and consent-aware. 2.2.5 Staff interact through Teams. Patient channels (SMS, and possibly web or voice) are not specified. 2.3 Non-functional requirements 2.3.1 Full audit of every access to PHI and every agent action. 2.3.2 High availability, with no stated numeric target. Proposed SLOs are in section 10. 2.3.3 Horizontal scalability across facilities. 2.4 Assumptions to validate [Assumption] 2.4.1 AWS is the only cloud, and Kubernetes means EKS. 2.4.2 Staff authenticate via Okta SSO. Patients may need a separate identity path. 2.4.3 The model provider is reachable under a signed HIPAA Business Associate Agreement (for example via Amazon Bedrock, which is HIPAA-eligible). The provider is not named in your inputs. 2.4.4 Epic access uses SMART on FHIR with backend service credentials, and write scope for appointments is available. 2.4.5 A small number of Postgres clusters is acceptable, rather than one per hospital. 2.5 Unresolved decisions [Open] 2.5.1 Model provider and model tiers. 2.5.2 Whether patients talk to agents directly, or only staff do. 2.5.3 Which languages must be supported. 2.5.4 Data residency and retention periods set by legal counsel. 2.5.5 Peak volumes: appointments per day, messages per hour, concurrent agent runs. 2.5.6 RTO and RPO targets. 2.5.7 Whether the vector database is managed (for example OpenSearch or pgvector) or a third-party product. 2.6 Regulatory note 2.6.1 HIPAA (Privacy Rule, Security Rule, Breach Notification Rule) is the regulation you named. Other obligations may apply depending on jurisdiction and services, such as state privacy laws, the TCPA for SMS, and 42 CFR Part 2 for substance-use records. Counsel should confirm which apply. Safeguards below are recommended engineering controls, not a legal determination. 3. Recommended Architecture and Rationale 3.1 Approaches evaluated 3.1.1 Full microservices: strong isolation but high operational cost, and distributed-transaction complexity that is unjustified before volumes are known. Rejected as the starting point. 3.1.2 Pure modular monolith: simplest to operate, but a single failure domain for PHI-heavy and externally dependent code. Insufficient alone. 3.1.3 Serverless-only: poor fit for long-running stateful agent workflows and for tight network controls around PHI. 3.1.4 Hybrid (recommended): modular monolith for orchestration, API, policy, and tool registry, plus separately deployed services where blast-radius or scaling needs differ. Event-driven messaging connects them. 3.2 Why hybrid 3.2.1 The agent control plane changes frequently and benefits from in-process calls and shared libraries. 3.2.2 The EHR gateway needs independent rate limiting, circuit breaking, and credentials isolation. 3.2.3 The notification service needs independent scaling, and its failures must not stall workflows. 3.2.4 Module boundaries inside the monolith are enforced by package structure and import rules, so a module can be extracted later. 3.3 Request path in words 3.3.1 A request arrives through the API gateway, and Okta-issued tokens are validated. 3.3.2 The control plane resolves tenant, role, and purpose, then starts or resumes a LangGraph run. 3.3.3 The supervisor agent chooses a worker agent. Workers call tools through a policy-enforcing tool layer. 3.3.4 Tools call the EHR gateway, scheduling adapters, retrieval service, or notification service. 3.3.5 Results return to the user via Teams, API response, or an asynchronous event. Every step emits audit and trace records. 3.4 Failure isolation 3.4.1 Separate Kubernetes namespaces and node groups for PHI-handling services. 3.4.2 Bulkheads: separate worker pools and queues per integration, so an Epic slowdown does not starve SMS. 3.4.3 Per-tenant concurrency and rate quotas stop one hospital from consuming shared capacity. 4. Core Components and Responsibilities 4.1 API gateway layer: TLS termination, WAF, request size limits, authentication handoff, and rate limiting per tenant and per user. 4.2 FastAPI application (control plane modules) 4.2.1 Identity and access module: token validation, tenant and role resolution, ABAC evaluation. 4.2.2 Orchestration module: LangGraph graphs, run management, checkpoints, cancellation. 4.2.3 Agent registry module: agent definitions, versions, allowed tools, and risk tiers. 4.2.4 Tool registry and policy module: schemas, scopes, validators, and approval rules. 4.2.5 Model gateway module: provider abstraction, routing, quotas, caching, and cost attribution. 4.2.6 Audit module: append-only event emission. 4.2.7 Memory and context module: state loading, retrieval calls, and redaction. 4.3 Separately deployed services 4.3.1 EHR/FHIR integration gateway: only component holding Epic credentials. Handles token refresh, FHIR resource mapping, throttling, retries, and circuit breakers. 4.3.2 Retrieval service: ingestion, embedding, and tenant-filtered search, isolated because it touches PHI-derived content. 4.3.3 Notification service: SMS and Teams delivery, consent checks, deduplication, delivery receipts. 4.3.4 Worker fleet: executes asynchronous runs consumed from queues. 4.4 Data stores: PostgreSQL (state, audit, registries), Redis (cache, locks, rate counters), S3 (documents, large artifacts, audit archives), and a vector index. 4.5 Messaging: a durable queue (Amazon SQS) for work and dead letters, plus an event stream or SNS topics for domain events. See section 13 for the choice. 4.6 Observability stack: OpenTelemetry collectors, metrics, log pipeline, and alerting. 5. Agent Orchestration and Execution Lifecycle 5.1 Agent set 5.1.1 Supervisor: classifies intent, enforces scope, and delegates. It holds no data tools. 5.1.2 Scheduling agent: search slots, book, reschedule, cancel. 5.1.3 Inquiry agent: answers administrative questions (hours, locations, preparation logistics from approved content) using RAG. 5.1.4 Discharge coordination agent: assembles checklists, creates tasks for staff, drafts patient-facing logistics for approval. 5.1.5 Record retrieval agent: fetches specific records for authorized staff and returns references, never summaries that interpret clinical meaning. 5.1.6 Reminder agent: schedules and sends follow-up reminders. 5.1.7 Clinical-boundary guard: a classifier step on input and output that routes clinical questions to a human queue. 5.2 Patterns 5.2.1 Supervisor-worker is the default. 5.2.2 Sequential flows for discharge (checklist, review, approval, send). 5.2.3 Parallel fan-out only for independent read-only lookups. 5.2.4 No free-form agent-to-agent chat. Agents communicate through typed state objects handed over by the supervisor, which keeps behavior auditable and bounded. 5.3 Lifecycle stages 5.3.1 Ingestion: authenticate, assign a run ID and idempotency key, attach tenant and role context. 5.3.2 Guarding: input checks for injection patterns and clinical-boundary classification. 5.3.3 Planning and routing: supervisor selects an agent and a bounded step budget. 5.3.4 Execution: model calls and tool calls, checkpointed after each node in Postgres. 5.3.5 Approval: high-risk actions pause and create an approval task in Teams. The run resumes on decision or expires on timeout. 5.3.6 Delivery: response returned or event emitted. 5.3.7 Closure: final audit record, cost attribution, and evaluation sampling. 5.4 Controls against uncontrolled behavior 5.4.1 Hard limits per run: max steps, max tool calls, max tokens, and wall-clock timeout. 5.4.2 Loop detection on repeated identical tool calls. 5.4.3 Cancellation endpoint and automatic cancel on user disconnect for non-critical runs. 5.4.4 Duplicate prevention through idempotency keys on runs and on every write tool. 5.4.5 Recovery: a worker crash resumes from the last checkpoint. Non-idempotent steps are marked so they are never blindly replayed. 5.5 Human approvals 5.5.1 Required for: sending patient messages beyond templated reminders, cancelling appointments on a patient’s behalf when ambiguous, and any record write. 5.5.2 Approvals are bound to a specific payload hash so approved content cannot change afterward. 5.5.3 [Open] The exact approval matrix needs sign-off from clinical operations and compliance. 6. Model, Tool, and Integration Infrastructure 6.1 Model gateway 6.1.1 A single internal interface hides provider differences, with structured output via schema validation and automatic repair or retry. 6.1.2 Routing by task tier: small, fast models for classification and extraction, stronger models for planning and drafting. 6.1.3 Streaming supported for staff-facing chat. Batch for background work. 6.1.4 Versions are pinned per agent version. Upgrades go through the evaluation gate in section 11. 6.1.5 Per-tenant and per-agent token quotas, with provider rate-limit handling via backoff and queueing. 6.1.6 Fallback: secondary model or provider for non-PHI-critical paths, and only if the secondary is also covered by a BAA. Otherwise degrade gracefully (see 10.4). 6.1.7 Caching limited to non-PHI content such as static FAQ answers. No PHI in shared caches. 6.1.8 Provider dependency is managed by keeping prompts, schemas, and evaluations provider-neutral. 6.2 Prompt-side data minimization 6.2.1 Send only the fields needed for the task. Replace identifiers with tokens where the task allows, and rehydrate after the model call. 6.2.2 No raw EHR dumps in prompts. 6.3 Tool framework 6.3.1 Every tool has a versioned schema, a declared risk tier (read, low-write, high-write), required scopes, and timeouts. 6.3.2 Invocation passes through a policy layer that checks: tenant match, user role, patient relationship, purpose, and approval requirement. The model never receives credentials. 6.3.3 Arguments are validated against schema, and patient identifiers are verified against the session context, not trusted from model output. 6.3.4 Outputs are normalized into a common envelope with status, data, and error class. 6.3.5 Lifecycle: registered, staged, enabled per tenant, deprecated, retired. 6.4 Integration specifics 6.4.1 Epic and FHIR: access exclusively through the integration gateway. Per-tenant credentials, request throttling aligned to Epic limits, caching of non-sensitive reference data, and circuit breakers. Mapping from FHIR resources to internal models lives in one module. 6.4.2 Scheduling systems: adapter per system behind a common interface. [Open] The number and type of scheduling systems is not specified, and Epic may itself be the scheduler at some sites. 6.4.3 Okta: OIDC for staff, group-to-role mapping, short-lived tokens. 6.4.4 Teams: bot for staff interaction and approvals, with tenant-scoped channels. Avoid placing PHI in Teams messages beyond what policy permits. 6.4.5 SMS: provider must offer a BAA if messages contain PHI. Default to minimal-content reminders with no clinical detail. 6.4.6 Webhooks: signed, replay-protected, and acknowledged quickly then processed asynchronously. 7. Data, State, Memory, and RAG 7.1 State categories 7.1.1 Session state: short-lived, in Redis with TTL, tenant-prefixed keys. 7.1.2 Workflow and checkpoint state: Postgres, durable, tied to run ID. 7.1.3 Execution records: Postgres, with large payloads in S3. 7.1.4 Transient state: in-process, never persisted. 7.2 Memory 7.2.1 Short-term: conversation window in checkpoint state, trimmed and summarized with PHI-aware redaction. 7.2.2 Long-term semantic memory: restricted to non-PHI organizational knowledge such as approved policies and FAQs. 7.2.3 Episodic memory about patients: [Rec] do not build agent-owned patient memory. Source patient facts from the EHR at request time, which avoids stale copies and a second PHI store. 7.2.4 Memory poisoning defense: writes to long-term memory require human or curated-pipeline approval, and every entry carries source and author. 7.3 Store selection 7.3.1 PostgreSQL: system of record for agent state, registries, audit index. Row-level security keyed on tenant. Strongly consistent. Encrypted with KMS. 7.3.2 Redis: cache, locks, rate limits. Treat as ephemeral. No durable PHI. 7.3.3 S3: documents and archives with per-tenant prefixes, bucket policies, SSE-KMS, versioning, and Object Lock for audit archives. 7.3.4 Vector store: [Rec] start with pgvector in the existing Postgres engine if corpus size permits, to cut one system. Move to OpenSearch if scale or latency demands. [Open] 7.3.5 Key strategy: per-tenant KMS keys where practical, to allow tenant-level cryptographic isolation and deletion. 7.4 Tenant isolation in data 7.4.1 Every table carries tenant ID, enforced by RLS and by application-layer checks. 7.4.2 Separate Postgres schemas or databases per tenant are an option if legal or contractual needs require it. [Open] Start with shared cluster plus RLS and strict testing. 7.4.3 Vector indexes either partitioned per tenant or filtered by mandatory tenant metadata, with tests proving filters cannot be omitted. 7.5 RAG pipeline 7.5.1 Corpus: approved administrative content only (policies, facility info, prep logistics, forms). Clinical content stays out of scope. 7.5.2 Ingestion: authenticated upload, malware scan, format normalization, owner and approval metadata. 7.5.3 Chunking: structure-aware (headings, sections), with overlap, and metadata for tenant, facility, document version, and effective dates. 7.5.4 Embeddings: provider-neutral interface, versioned, so re-embedding is a controlled migration. 7.5.5 Retrieval: hybrid keyword plus vector, with mandatory tenant and facility filters. 7.5.6 Reranking: a lightweight reranker on the top candidates. 7.5.7 Context assembly: cap tokens, order by relevance, include citations with document ID and version. 7.5.8 Grounding: answers must cite retrieved sources. If retrieval confidence is low, the agent says it cannot answer and offers a human handoff. 7.5.9 Source validation: only approved, in-date documents are indexed, and expired documents are deindexed automatically. 7.5.10 Retrieved text is treated as untrusted input (see 9.4). 7.6 Retention and lifecycle 7.6.1 [Open] Retention periods for transcripts, audit logs, and checkpoints must come from legal and compliance. 7.6.2 Recommended: store the minimum, expire transient state aggressively, and make deletion by tenant and by patient technically possible. 7.6.3 Backups: automated, encrypted, cross-region copies, with restore drills. 8. API and Communication Architecture 8.1 Protocol choices 8.1.1 REST with JSON for client-facing APIs, versioned in the path. 8.1.2 Server-sent events for streaming agent progress to staff UIs. WebSockets only if bidirectional needs appear. 8.1.3 Internal service calls via REST or gRPC. [Rec] REST initially for simplicity. gRPC for the EHR gateway if latency or schema discipline warrants. 8.1.4 Queues for asynchronous work. Event streaming for domain events. 8.1.5 GraphQL is not recommended, as it adds complexity without a clear need here. 8.2 API conventions 8.2.1 Authentication via Okta-issued OAuth tokens. Service-to-service via short-lived workload identities (IAM roles for service accounts). 8.2.2 Strict input validation with Pydantic models and size limits. 8.2.3 Cursor-based pagination. 8.2.4 Long-running operations return 202 with a run ID and a status endpoint. 8.2.5 Idempotency-Key header required on all mutating endpoints. 8.2.6 Consistent error envelope with machine-readable codes and no PHI in error text. 8.2.7 Backward-compatible changes within a version. Deprecation windows announced. 8.3 Messaging semantics 8.3.1 At-least-once delivery with idempotent consumers, because exactly-once is not realistic. 8.3.2 Deduplication by message ID and business key. 8.3.3 Ordering: per-patient or per-appointment ordering through FIFO queue groups where order matters (for example reschedule then cancel). 8.3.4 Dead-letter queues per consumer, with alerts, a triage runbook, and safe redrive tooling. 8.3.5 Outbox pattern: state changes and event publication committed in one Postgres transaction, then relayed to the queue. 8.4 Webhooks and inbound events 8.4.1 Verify signatures and timestamps. 8.4.2 Persist raw event first, acknowledge, then process. 8.4.3 Replay protection and rate limiting. 9. Security, Privacy, and Governance 9.1 Identity and access 9.1.1 Staff authenticate through Okta with MFA. Roles are derived from Okta groups. 9.1.2 RBAC for coarse permissions (scheduler, nurse coordinator, records clerk, admin). 9.1.3 ABAC for fine-grained decisions: tenant, facility, patient relationship (care-team or department), purpose of use, and time. 9.1.4 Agents act with delegated, down-scoped identity: an agent never has more permission than the invoking user, and background agents use narrow service identities. 9.1.5 Break-glass access, if ever needed, is separate, time-limited, and heavily audited. [Open] 9.2 Tenant isolation 9.2.1 Tenant context established at the edge and propagated immutably. 9.2.2 Enforced at each layer: API, tool policy, database RLS, cache key prefixes, vector filters, S3 prefixes, and per-tenant keys. 9.2.3 Automated cross-tenant access tests run in CI and in staging continuously. 9.3 Encryption and secrets 9.3.1 TLS everywhere, including in-cluster (service mesh mTLS is optional). 9.3.2 Encryption at rest with KMS for Postgres, S3, Redis, and the vector store. 9.3.3 Secrets in AWS Secrets Manager, injected at runtime, rotated, and never visible to models or logs. 9.4 AI-specific threats and mitigations 9.4.1 Prompt injection (direct and via retrieved or EHR content): treat all external text as data, separate instructions from data in prompts, restrict tools by task, and verify identifiers outside the model. 9.4.2 Data exfiltration: egress restrictions, no tool that sends arbitrary content to arbitrary destinations, output scanning for PHI patterns where policy forbids it, and recipient allow-lists for messages. 9.4.3 Malicious or unsafe tools: only registered, reviewed tools. No dynamic code execution tools. 9.4.4 Privilege escalation: policy checks outside the model, scopes bound to the user, and approval gates. 9.4.5 Memory poisoning: curated writes only (see 7.2.4). 9.4.6 Excessive autonomy: bounded budgets, risk-tiered approvals, and kill switches per agent and per tenant. 9.4.7 Sensitive-data exposure in logs and traces: structured redaction at the logging layer, and PHI-free metrics. 9.5 Privacy and governance 9.5.1 Data minimization in prompts, storage, and logs. 9.5.2 Consent: patient communication preferences and consent recorded and checked before SMS or outreach. [Open] Consent source of truth. 9.5.3 Audit trail: who, what, which patient, which tenant, which tool, which agent version, what outcome, and the approval chain. Append-only, tamper-evident (hash chaining or Object Lock), and retained per policy. 9.5.4 Governance: an AI review board for agent changes, a model and tool change log, vendor BAAs tracked, periodic access reviews, and incident response with breach-assessment procedures. Counsel confirms regulatory specifics. 10. Reliability, Performance, and Scalability 10.1 Fault tolerance 10.1.1 Retries with exponential backoff and jitter, only on idempotent or keyed operations. 10.1.2 Circuit breakers on Epic, scheduling systems, model providers, and SMS. 10.1.3 Bulkheads via separate queues, pools, and per-tenant limits. 10.1.4 Checkpointed workflows resume after pod loss. 10.1.5 Multi-AZ for EKS, RDS, Redis, and the queue. Cross-region backup, and warm standby if RTO requires. [Open] 10.2 Proposed SLIs and SLOs (starting points to be ratified) [Assumption] 10.2.1 API availability: 99.9% monthly for staff-facing endpoints. 10.2.2 Synchronous agent response: p95 under a target agreed with users, for example under 8 seconds for simple scheduling queries. 10.2.3 Reminder delivery: 99.5% of reminders sent within a defined window of their scheduled time. 10.2.4 Tool success rate: above 99% excluding upstream outages, tracked per tool. 10.2.5 Zero confirmed cross-tenant data exposures, with any event a severity-one incident. 10.2.6 Audit completeness: 100% of PHI accesses have a matching audit record, verified by reconciliation jobs. 10.3 Performance and scaling 10.3.1 Likely bottlenecks: model latency and rate limits, Epic API throughput, Postgres write load from checkpoints and audit, and queue backlog during bursts such as batch reminders. 10.3.2 Scale stateless API and workers horizontally with autoscaling on queue depth and latency. 10.3.3 Postgres: connection pooling, partitioning for audit and checkpoint tables by time, read replicas for reporting, and archival to S3. 10.3.4 Cache reference data (facility info, provider lists) with short TTLs. Never cache PHI in shared keys. 10.3.5 Spread batch work (reminder generation) over time to avoid synchronized spikes. 10.3.6 Capacity planning waits on the volumes listed in 2.5.5. Load tests establish real limits. 10.4 Graceful degradation 10.4.1 Model outage: fall back to templated flows for reminders and simple scheduling, and queue the rest with user notice. 10.4.2 Epic outage: serve cached non-sensitive info, queue writes with clear user messaging, and never fabricate availability. 10.4.3 SMS outage: retry, then alert staff for critical reminders. 10.5 Single points of failure to eliminate: single-AZ databases, single queue consumers, a single secrets path, and one EHR gateway replica. 11. Observability and Agent Evaluation 11.1 Logging, tracing, metrics 11.1.1 Structured JSON logs with run ID, tenant ID, agent version, and tool name, and no PHI. 11.1.2 Distributed tracing with OpenTelemetry across API, workers, gateway, model calls, and tool calls. 11.1.3 Metrics: latency, error rates, queue depth, retries, circuit breaker state, token usage, cost per task, approval wait times, and DLQ counts. 11.1.4 Alerts tied to SLO burn rates, not raw thresholds. Separate security alerts for anomalous access patterns. 11.1.5 Incident management: on-call rotation, runbooks per component, severity definitions, and blameless post-incident reviews. 11.2 Agent evaluation dimensions 11.2.1 Task completion rate and correctness of final state (for example, was the right appointment booked). 11.2.2 Tool selection accuracy and argument correctness. 11.2.3 Retrieval quality (recall and precision on labeled questions). 11.2.4 Grounding and hallucination rate: answers unsupported by cited sources. 11.2.5 Policy violations: clinical advice given, scope breaches, PHI in wrong place, and approval bypasses. Target is zero. 11.2.6 Latency, reliability, and cost per completed task, tracked by tenant and agent. 11.3 Evaluation methods 11.3.1 Curated datasets built from de-identified or synthetic scenarios per agent, including adversarial cases such as injection attempts and clinical-boundary probes. 11.3.2 Automated regression suites run on every prompt, model, tool, or agent change, with release gates. 11.3.3 Production sampling with human review of a defined percentage of runs, plus staff feedback buttons. 11.3.4 Drift monitoring: compare live metrics to baseline after model or provider changes. 11.3.5 Findings feed back into prompts, tools, retrieval content, and datasets. 12. Deployment and Infrastructure Strategy 12.1 Containers and orchestration: Docker images built from minimal bases, signed and scanned. Amazon EKS with separate namespaces for control plane, PHI services, and workers, plus network policies denying traffic by default. 12.2 Infrastructure as code: Terraform for AWS resources, Helm or Kustomize for cluster workloads, and GitOps (Argo CD or equivalent) for promotion. Specific tooling is a recommendation. 12.3 Environments 12.3.1 Development: synthetic data only. 12.3.2 Test and staging: synthetic or properly de-identified data, mirrors production topology, runs tenant-isolation and load tests. 12.3.3 Production: separate AWS accounts, restricted access, change control. 12.3.4 No real PHI outside production unless explicitly approved and protected equally. 12.4 CI/CD 12.4.1 Pipeline stages: lint, type check, unit tests, contract tests, security scans (dependencies, containers, secrets), integration tests, evaluation suite, and deployment. 12.4.2 Release safety: canary or progressive rollout per tenant (start with one low-risk hospital), feature flags per agent and tool, and automatic rollback on SLO regression. 12.4.3 Database migrations: backward-compatible, expand-then-contract, and tested on production-like data volumes. 12.5 Configuration and secrets: environment-specific config in versioned repositories, secrets via Secrets Manager, and per-tenant settings in the database with audit. 12.6 Backup and disaster recovery: automated Postgres snapshots and point-in-time recovery, S3 versioning and cross-region replication, and tested restore and failover drills. [Open] RTO and RPO values. 12.7 Operational ownership: platform team for infrastructure, an agent engineering team for orchestration and evaluation, a security and compliance function for policy, and clinical operations as owner of agent scope and approval matrices. [Assumption] Team structure is not specified. 13. Technology Recommendations and Trade-Offs 13.1 Python and FastAPI [Fact]: fits the ecosystem, with strong async support and typed validation. Limitation: careful concurrency management is needed around blocking calls. 13.2 LangGraph [Fact]: durable, checkpointed graph execution suits approvals and recovery. Trade-off: the framework is evolving, so pin versions and keep business logic outside graph definitions to limit lock-in. 13.3 Model access 13.3.1 [Rec] Prefer a HIPAA-eligible managed route such as Amazon Bedrock if it offers suitable models, because it keeps traffic inside AWS and simplifies BAAs and network controls. 13.3.2 Alternative: a direct provider API under a BAA. Trade-off: broader model choice, but another vendor and egress path. 13.3.3 [Open] Final choice depends on model quality tests and contract terms. 13.4 PostgreSQL (RDS or Aurora): ACID, row-level security, and JSONB flexibility. Alternative: DynamoDB for some state, rejected for added complexity and weaker relational and RLS support. 13.5 Redis (ElastiCache): speed for sessions and rate limits. Do not treat as durable. 13.6 Vector store: pgvector first, OpenSearch later if needed. Trade-off: pgvector has fewer features and scaling limits, but fewer systems and a single isolation model. 13.7 Messaging: SQS for work queues (simple, managed, DLQ built in) and SNS or EventBridge for events. Kafka or MSK only if event-stream replay and very high throughput become necessary. Avoid adding it prematurely. 13.8 Auth: Okta [Fact] for identity, AWS IAM and workload identity for services. 13.9 Observability: OpenTelemetry for portability, with a managed backend (CloudWatch plus a tracing and metrics platform of your choice). Vendor choice is open. 13.10 Not recommended: a service mesh initially, GraphQL, per-hospital full stacks, and an agent framework beyond LangGraph. Each adds cost without a demonstrated need. 14. Implementation Roadmap Dates, staffing, and budgets are intentionally omitted. 14.1 Phase 1: Foundations 14.1.1 Objectives: secure platform, identity, tenancy model, audit backbone. 14.1.2 Activities: AWS accounts and networking, EKS, Terraform, Okta integration, tenant context propagation, Postgres with RLS, audit module, CI/CD, observability baseline. 14.1.3 Dependencies: legal and compliance input on retention and BAAs, Okta access. 14.1.4 Deliverables: working skeleton service with tenant-aware auth and audit. 14.1.5 Completion criteria: cross-tenant isolation tests pass, audit events verified, and deployments reproducible. 14.2 Phase 2: Integration gateway 14.2.1 Objectives: reliable, safe Epic and FHIR access. 14.2.2 Activities: gateway service, credentials handling, resource mapping, throttling, circuit breakers, sandbox testing, contract tests. 14.2.3 Dependencies: Epic access and sandbox, scheduling system documentation. 14.2.4 Deliverables: read-only then write-capable adapters. 14.2.5 Completion criteria: contract tests pass, failure injection shows graceful behavior, and rate limits respected. 14.3 Phase 3: Orchestration and tool framework 14.3.1 Objectives: bounded, durable agent execution. 14.3.2 Activities: LangGraph runs with Postgres checkpoints, tool registry and policy layer, model gateway, approval flow via Teams, run limits, and cancellation. 14.3.3 Dependencies: Phase 1, and Phase 2 for real tools (mocks until then). 14.3.4 Deliverables: supervisor plus one agent end to end. 14.3.5 Completion criteria: crash-recovery and idempotency tests pass, and policy denies unauthorized calls. 14.4 Phase 4: First agents (scheduling and reminders) 14.4.1 Objectives: deliver value on structured, low-risk workflows. 14.4.2 Activities: scheduling agent, reminder agent, notification service, consent checks, evaluation datasets. 14.4.3 Dependencies: Phases 2 and 3, and SMS provider BAA. 14.4.4 Completion criteria: evaluation thresholds met, pilot at one hospital with progressive rollout. 14.5 Phase 5: Data, RAG, and inquiry agent 14.5.1 Activities: ingestion pipeline, retrieval service, grounding checks, inquiry agent, clinical-boundary guard. 14.5.2 Completion criteria: retrieval and grounding targets met, and zero policy violations on adversarial sets. 14.6 Phase 6: Record retrieval and discharge coordination 14.6.1 Rationale: highest PHI and workflow sensitivity, so they come after controls are proven. 14.6.2 Activities: relationship-based ABAC, retrieval agent, discharge workflow with approvals. 14.6.3 Completion criteria: security review passed, approval matrix signed off, and a pilot with staff feedback. 14.7 Phase 7: Hardening and scale 14.7.1 Activities: load and fault testing, DR drills, cost attribution, tuning, and network-wide rollout by facility group. 14.7.2 Completion criteria: SLOs met under load and recovery objectives demonstrated. 14.8 Phase 8: Continuous optimization: evaluation-driven improvement, model upgrades through gates, and periodic security and access reviews. 15. Testing and Production Readiness 15.1 Test types 15.1.1 Unit and API tests, including validation edge cases. 15.1.2 Contract tests against Epic, FHIR, scheduling, Okta, Teams, and SMS interfaces. 15.1.3 Integration and database tests, including migrations and RLS. 15.1.4 Workflow tests: checkpoint resume, cancellation, timeouts, approval expiry. 15.1.5 Tool tests: schema, authorization, idempotency, error mapping. 15.1.6 Model and retrieval tests via the evaluation suite. 15.1.7 Authentication and authorization tests across role and attribute combinations. 15.1.8 Security tests: injection, exfiltration, privilege escalation, dependency and container scans, and external penetration testing. 15.1.9 Performance, load, and concurrency tests, including burst reminder scenarios. 15.1.10 Fault injection: provider outages, Epic latency, queue failures, pod and AZ loss. 15.1.11 Recovery tests: backup restore and failover. 15.1.12 Regression tests on every change. 15.1.13 Tenant isolation tests: attempts to read or write across tenants at every layer. 15.2 Release gates and thresholds [Assumption: values to be ratified] 15.2.1 No critical or high security findings open. 15.2.2 Zero cross-tenant leaks in isolation tests. 15.2.3 Policy-violation rate of zero on adversarial sets. 15.2.4 Task completion, grounding, and tool accuracy above agreed thresholds per agent. 15.2.5 SLO targets met in load tests. 15.2.6 Rollback tested. 15.3 Production readiness checklist: BAAs signed with every PHI-touching vendor, runbooks and on-call in place, alerts verified, DR drill completed, audit reconciliation running, kill switches tested, compliance and clinical sign-off, and a staged rollout plan. 16. Risk and Failure Assessment (ordered by priority) 16.1 Cross-tenant or unauthorized PHI exposure 16.1.1 Cause: filter omission, cache key mistakes, over-broad service identity. 16.1.2 Impact: regulatory, legal, and reputational harm. Likelihood: medium without controls. Severity: critical. 16.1.3 Detection: isolation tests, anomaly alerts, audit reconciliation. 16.1.4 Mitigation: layered enforcement (9.2), per-tenant keys, mandatory test gates. 16.1.5 Residual: low but non-zero, so keep continuous testing. 16.2 Prompt injection causing unauthorized actions or exfiltration 16.2.1 Cause: untrusted text from users, documents, or EHR fields. 16.2.2 Impact: wrong actions or leaked data. Likelihood: high exposure. Severity: high to critical. 16.2.3 Detection: adversarial evaluation, tool-call anomaly monitoring. 16.2.4 Mitigation: policy outside the model, narrow tools, approvals, recipient allow-lists. 16.2.5 Residual: medium. It cannot be fully eliminated, so keep autonomy bounded. 16.3 Agents drifting into clinical territory 16.3.1 Cause: ambiguous user questions and model overreach. 16.3.2 Impact: patient safety. Severity: critical. 16.3.3 Detection: boundary classifier logs, review sampling. 16.3.4 Mitigation: scope limits, guard on input and output, human handoff, templated responses for sensitive topics. 16.3.5 Residual: low to medium. 16.4 Epic and scheduling integration failures or rate limits 16.4.1 Impact: failed bookings, stale data, duplicate appointments. Likelihood: high. Severity: medium to high. 16.4.2 Mitigation: gateway with throttling, idempotency, reconciliation jobs, graceful degradation. 16.4.3 Residual: medium. 16.5 Model provider outage, quota exhaustion, or silent quality change 16.5.1 Mitigation: fallbacks, version pinning, drift monitoring, templated degradation. 16.5.2 Residual: medium, because of dependency on the provider. 16.6 Duplicate or conflicting actions (double bookings, repeated SMS) 16.6.1 Mitigation: idempotency keys, outbox, deduplication, ordered queues. 16.6.2 Residual: low. 16.7 Postgres contention and growth from checkpoints and audit 16.7.1 Mitigation: partitioning, archival, pooling, monitoring. 16.7.2 Residual: low to medium. 16.8 Queue backlogs and DLQ neglect 16.8.1 Mitigation: autoscaling on depth, alerts, runbooks, redrive tooling. 16.9 Poor observability or inadequate evaluation 16.9.1 Impact: silent quality or safety degradation. Mitigation: SLO alerts, evaluation gates, production sampling. 16.10 Compliance gaps from vendors lacking BAAs or unclear data handling: verify before integration, and keep a vendor register. 16.11 Cost overrun from token usage: per-tenant quotas, model tiering, cost-per-task dashboards, and budget alerts. 16.12 Overengineering or delayed delivery: mitigated by the phased roadmap and the modular monolith start. 17. Final Recommendations and Priorities 17.1 First priorities 17.1.1 Settle the open decisions in 2.5, especially model provider and BAA, patient-facing scope, and retention. 17.1.2 Build tenant isolation, audit, and policy enforcement before any agent logic. 17.1.3 Establish the integration gateway early, since it determines feasibility. 17.2 Design principles to hold throughout 17.2.1 Agents advise and assist on administration. Humans decide anything clinical. 17.2.2 Authorization is enforced outside the model, always. 17.2.3 Minimize PHI in prompts, memory, logs, and caches. 17.2.4 Every write is idempotent, audited, and, when risky, approved. 17.2.5 Start simple (modular monolith, pgvector, SQS) and extract or upgrade only on evidence. 17.3 Rollout strategy: begin with scheduling and reminders at one hospital, expand by facility group, and introduce record retrieval and discharge coordination only after controls and evaluations are proven. 17.4 Information that would sharpen this blueprint: traffic volumes, Epic integration details and scopes, the list of scheduling systems, patient-channel scope, RTO and RPO, the legal retention matrix, and the approved model and vendor list.