# Production-Grade Enterprise RAG + Knowledge Graph Architecture ## Healthcare Medical Records & Clinical Notes ### Principal Enterprise AI Architect Design --- # High-Level Architecture ┌────────────────────────────┐ │ Hospital Information Systems│ │ EMR / EHR / PACS / HL7/FHIR │ └──────────────┬──────────────┘ │ Data Ingestion Layer │ ┌─────────────────────────────┼────────────────────────────┐ │ │ │ PDFs/Scans Clinical Notes Structured Data │ │ │ OCR + Parsing Text Cleaning FHIR Parser │ │ │ └─────────────────────────────┼────────────────────────────┘ │ Semantic Chunking │ Metadata Enrichment Pipeline │ ┌────────────────────────┴───────────────────────┐ │ │ OpenAI Embeddings Neo4j KG Builder text-embedding-3-large Entities + Relationships │ │ │ │ Qdrant Vector DB Neo4j Knowledge Graph │ │ └──────────────────────┬─────────────────────────┘ │ Hybrid Retrieval Engine │ BM25 + Dense Search + Graph Traversal │ Cohere Rerank v3 │ Context Compression Layer │ GPT-4o Generation │ Clinical AI Assistant API --- # 1. Data Ingestion & Chunking Pipeline ## Supported Sources - Electronic Health Records (EHR) - Electronic Medical Records (EMR) - Clinical Notes - Radiology Reports - Pathology Reports - Discharge Summaries - Laboratory Reports - Insurance Documents - Prescriptions - Scanned PDFs - HL7 Messages - FHIR Resources --- ## Parsing Layer ### PDF Parsing Use: - PyMuPDF - Unstructured.io - LlamaParse - Azure Document Intelligence Extract: - Paragraphs - Tables - Headers - Footnotes - Images - Captions - Signatures --- ## OCR Pipeline If scanned: Image ↓ Deskew ↓ Noise Removal ↓ OCR (Tesseract / Azure OCR) ↓ Layout Detection ↓ Semantic Reconstruction --- ## Table Parsing Instead of flattening tables: Patient Lab Results | Date | Test | Result | |------|------|--------| |12 Jan|HbA1c|8.2| Convert into structured JSON: { Date, Test, Result } Each row becomes searchable. --- ## Semantic Chunking Strategy Never fixed chunking. Instead: Clinical Sections ↓ SOAP Notes ↓ Diagnosis ↓ Medication ↓ History ↓ Assessment ↓ Plan ↓ Lab Reports ↓ Radiology Findings Chunk Size 600–900 tokens Overlap 100–150 tokens Boundary Detection Sentence Transformer Section Headers Medical Ontology --- ## Metadata Tagging Each chunk stores: Patient ID Encounter ID Hospital ID Doctor Department Specialty Document Type Visit Date Admission Date Diagnosis Codes (ICD-10) SNOMED Codes LOINC Codes Medication Language Source Version Tenant ID ACL Sensitivity Level Embedding Version Chunk Hash --- # 2. Hybrid Retrieval Architecture ## Multi-Stage Retrieval User Query ↓ Query Rewriting ↓ Intent Detection ↓ Hybrid Search ↓ Graph Expansion ↓ Reranking ↓ Context Compression ↓ GPT-4o --- ## Dense Retrieval Embedding: OpenAI text-embedding-3-large Stored in: Qdrant Similarity: Cosine Top K: 50 --- ## Sparse Retrieval BM25 Useful for: Medical Codes Drug Names Lab Values Dates Rare Diseases Acronyms Top K 100 --- ## Reciprocal Rank Fusion (RRF) Combine BM25 + Dense Search ↓ Unified Candidate Set ↓ Top 100 --- ## GraphRAG Neo4j stores Patient ↓ Encounter ↓ Diagnosis ↓ Medication ↓ Doctor ↓ Lab Test ↓ Radiology ↓ Procedure ↓ Allergy ↓ Hospital Relationships Patient HAS_VISIT Encounter DIAGNOSED_WITH Disease PRESCRIBED Medication ORDERED Lab Test REFERRED_TO Doctor --- ## Context Stitching Example Question "Why was Metformin discontinued?" Retrieve Clinical Note + Medication History + Lab Results + Doctor Assessment + Follow-up Visit ↓ Graph joins related entities ↓ Unified Clinical Context --- # 3. Reranking & Context Compression Candidate Documents ↓ Cohere Rerank v3 ↓ Score ↓ Top 15 ↓ Compression ↓ GPT-4o --- ## Relevance Scoring Final Score = 0.45 Dense Similarity + 0.30 BM25 + 0.15 Graph Connectivity + 0.10 Metadata Match --- ## Context Compression Remove Duplicate paragraphs Old versions Repeated lab reports Boilerplate Headers Footers Legal disclaimers Only retain Clinical Findings Diagnosis Medications Assessment Plan Evidence --- ## Dynamic Context Window Available Tokens ↓ Ranked Chunks ↓ Graph Context ↓ Summaries ↓ Final Prompt Keep within 128K context --- # 4. Security, Access Control & Privacy ## Authentication OIDC OAuth2 Azure AD Okta JWT --- ## Authorization RBAC Doctor Nurse Researcher Admin Patient ABAC Department Hospital Region Emergency Access --- ## Row-Level Security (RLS) Every record contains Tenant_ID Hospital_ID Department Patient_ID Doctor_ID Access Query WHERE Tenant_ID = CurrentTenant AND Department = CurrentDepartment AND Role IN AllowedRoles --- ## Multi-Tenant Isolation Hospital A ↓ Separate Namespace ↓ Separate Qdrant Collection ↓ Separate Neo4j Subgraph ↓ Separate Encryption Keys Hospital B ↓ Same Architecture ↓ No Cross-Tenant Access --- ## Encryption At Rest AES-256 In Transit TLS 1.3 Secrets HashiCorp Vault Azure Key Vault AWS KMS --- ## PII Masking Mask Patient Name Phone Address Email SSN Insurance Number Medical Record Number Before LLM ↓ Named Entity Recognition ↓ Replace John Smith ↓ PATIENT_NAME --- ## Audit Logging Store User Timestamp Query Retrieved Documents Generated Answer LLM Version Latency Prompt Hash --- # 5. Evaluation & Monitoring ## RAG Triad ### Context Relevance Did retrieval fetch correct clinical evidence? Metrics Recall@K Precision@K MRR NDCG --- ### Groundedness Did GPT answer only using retrieved evidence? Metrics Faithfulness Citation Coverage Hallucination Rate Grounding Score --- ### Answer Relevance Does answer satisfy physician question? Metrics BLEU ROUGE LLM Judge Human Review Clinical Accuracy --- ## Operational Monitoring Track Embedding Latency Retrieval Latency Graph Query Latency Rerank Latency LLM Latency Total End-to-End Latency Error Rate Cache Hit Rate Token Usage GPU Utilization Cost Per Query --- ## Latency Guardrails Embedding <100 ms Vector Search <80 ms BM25 <50 ms Neo4j Traversal <120 ms Rerank <150 ms LLM Generation <2.5 sec Total <3.5 sec --- # Production Technology Stack | Layer | Technology | |--------|------------| |API Gateway|Kong / NGINX| |Authentication|OAuth2 + Azure AD| |Document Parsing|LlamaParse, Unstructured| |OCR|Azure OCR, Tesseract| |Chunking|Semantic Chunker| |Embeddings|OpenAI text-embedding-3-large| |Vector Database|Qdrant| |Keyword Search|BM25 (Elasticsearch/OpenSearch)| |Knowledge Graph|Neo4j| |Graph Retrieval|Cypher + GraphRAG| |Reranking|Cohere Rerank v3| |LLM|GPT-4o| |Caching|Redis| |Workflow|LangGraph| |Observability|Langfuse, OpenTelemetry, Prometheus, Grafana| |Evaluation|RAGAS, TruLens, DeepEval| |Secrets|HashiCorp Vault| |Deployment|Kubernetes| |CI/CD|GitHub Actions| |Monitoring|Grafana + Prometheus| --- # Enterprise Query Flow 1. User submits a clinical question. 2. Query is normalized and rewritten. 3. Hybrid retrieval executes BM25 + Qdrant vector search. 4. Neo4j GraphRAG expands related medical entities. 5. Reciprocal Rank Fusion merges dense and sparse results. 6. Cohere Rerank v3 reorders candidates by semantic relevance. 7. Context compression removes redundant and low-value content. 8. GPT-4o generates a grounded response using only retrieved evidence. 9. PII masking and access control are enforced before generation. 10. Answer, citations, audit logs, and evaluation metrics are recorded for monitoring and continuous improvement. --- # Key Enterprise Design Principles - Semantic chunking over fixed-size chunking - Hybrid retrieval (Dense + Sparse + Graph) - Metadata-rich indexing - Knowledge graph–based context expansion - Multi-stage reranking and context compression - Strict row-level security and tenant isolation - End-to-end PII protection and encryption - Continuous RAG evaluation using Context Relevance, Groundedness, and Answer Relevance - Production-grade observability, latency guardrails, and scalable Kubernetes deployment - Healthcare compliance readiness (HIPAA/GDPR aligned design)