Temporal RAG Architecture for Enterprise Knowledge Management ## Assumptions - Sources: SharePoint, Confluence, Google Drive, S3, and Git wikis, with PDF, DOCX, HTML, and Markdown files - Roughly 5M documents, 50 to 200 queries/sec at peak, multi-tenant with per-document ACLs - Temporal (self-hosted or Cloud) orchestrates **ingestion only**. The query path stays synchronous for low latency. - Vector DB is Qdrant or pgvector, with OpenSearch for BM25 hybrid search. The LLM is behind an internal gateway. - SSO via OIDC (Okta/Entra ID) ## System Architecture ```mermaid flowchart TB subgraph SRC[Sources] S1[SharePoint / Confluence] S2[Drive / S3 / Git] end subgraph ING[Ingestion Control Plane - Temporal] CDC[Change Webhooks / Pollers] --> Q[(Kafka / SQS)] Q --> STARTER[Workflow Starter] STARTER --> TS[Temporal Server] TS --> WF[IngestDocumentWorkflow] WF --> ACT[Activity Workers] end subgraph DP[Data Plane - Activities] A1[Fetch + Hash Check] A2[Parse / OCR] A3[Clean + Chunk] A4[Embed Batch] A5[Upsert Vectors] A6[Index BM25 + Metadata] end ACT --> A1 --> A2 --> A3 --> A4 --> A5 --> A6 S1 & S2 --> CDC A1 -. read .-> S1 & S2 A1 & A2 --> OBJ[(Object Store: raw + parsed)] A4 --> EMB[Embedding Service] A5 --> VDB[(Vector DB)] A6 --> SEARCH[(OpenSearch BM25)] A6 --> META[(Postgres: metadata, ACLs, doc versions)] subgraph QP[Query Plane - Synchronous] U[User / App] --> GW[API Gateway + AuthN/Z] GW --> QS[RAG Query Service] QS --> QR[Query Rewrite / Guardrails] QR --> RET[Hybrid Retriever] RET --> VDB RET --> SEARCH RET --> META RET --> RR[Reranker] RR --> CTX[Context Builder + Citations] CTX --> LLM[LLM Gateway] LLM --> OUT[Output Guardrails] OUT --> GW QS <--> CACHE[(Redis Cache)] end subgraph OBS[Observability] O1[OpenTelemetry] --> O2[Prometheus / Grafana] O1 --> O3[Tracing: Tempo/Jaeger] O4[LLM Eval / Feedback Store] end TS & ACT & QS & LLM -.-> O1 ``` ## Temporal vs. RAG: The Boundary | Layer | Owns | Does **not** own | |---|---|---| | **Temporal (orchestration)** | Durable state, retries, timeouts, fan-out, backfills, re-embedding, scheduling, human approval steps | Retrieval, ranking, prompts | | **RAG (retrieval + generation)** | Query understanding, hybrid search, reranking, context assembly, LLM calls, citations | Long-running jobs, retry state | Temporal runs the pipeline that builds the index. The RAG services read from it. Keeping these separate prevents a slow ingestion backlog from affecting query latency. ## Temporal Design **Workflows** (deterministic, no I/O): - `IngestDocumentWorkflow(docId, version)`: fetch, parse, chunk, embed, upsert, index. Workflow ID is `doc:{id}:{contentHash}`, which gives idempotency and dedup. - `SourceSyncWorkflow` (Schedule-triggered): lists changes and starts child workflows in bounded batches. - `ReindexWorkflow`: re-embeds everything after a model or chunking change. It uses `continueAsNew` every N documents to keep history small. - `DeleteDocumentWorkflow`: removes vectors, BM25 entries, and metadata, honoring the source's deletion events. **Activities** (side effects, retried): | Activity | Timeout | Retry policy | |---|---|---| | FetchDocument | 2 min | Exponential, 5 attempts. Non-retryable on 403/404. | | ParseDocument (OCR) | 10 min + heartbeat | 3 attempts. Heartbeat for long OCR jobs. | | ChunkDocument | 1 min | 2 attempts. Pure compute. | | EmbedBatch | 1 min | Backoff with jitter on 429. | | UpsertVectors | 1 min | Idempotent upsert keyed by `chunkId`. | | UpdateMetadata | 30 sec | Idempotent transaction. | **Key mechanisms** - **State:** Event history is persisted, so workers can crash and resume exactly where they left off. - **Failures:** Activities retry per policy. Unrecoverable failures go to a dead-letter workflow with an alert, and the document is marked `FAILED` in Postgres. - **Consistency:** Use a saga pattern. If upsert succeeds but metadata fails, compensate by deleting the new vectors. Use `version` fields so queries only see fully indexed versions (swap the `active_version` pointer last). - **Backpressure:** Task queue separation (`parse-ocr`, `embed`, `io`) with per-queue worker scaling and `maxConcurrentActivityExecutions`. - **Signals and queries:** Signal to cancel or prioritize. Query to report progress to an admin UI. ## End-to-End Data Flow **Ingestion** 1. A webhook or poll detects a change and publishes it to the queue. 2. The starter launches `IngestDocumentWorkflow`. A content hash match makes the run a no-op. 3. The document is fetched with its ACLs. Raw files are stored in the object store. 4. Parsing handles layout, tables, and OCR. Parsed output is stored. 5. Chunking is structure-aware, around 400 to 800 tokens with 10 to 15% overlap and a heading-path prefix. 6. Embeddings are generated in batches of 64 to 128. 7. Vectors, with payload (`docId`, `tenant`, `aclGroups`, `version`, `sourceUrl`, `updatedAt`), are upserted to the vector DB. BM25 and metadata are written in parallel. 8. The `active_version` pointer swaps and old chunks are garbage-collected. **Query** 1. The gateway authenticates the user and resolves group memberships. 2. The query is rewritten, and the cache is checked. 3. Hybrid retrieval runs vector and BM25 search with **ACL filters applied in the DB query**, then fuses results with RRF (top 50). 4. A cross-encoder reranker narrows the results to the top 6 to 10. 5. The context builder deduplicates, orders, and budgets tokens, and attaches citation IDs. 6. The LLM generates a grounded answer with an instruction to cite sources or say "not found". 7. Output guardrails run (PII, citation validity), and the response streams to the user. ## Component Notes - **Parser:** Apache Tika or Unstructured, plus OCR. This is the most failure-prone step, so isolate it in its own worker pool. - **Embedding service:** Self-hosted or API. Pin the model version and store it on each chunk so mixed versions can't be queried together. - **Postgres:** Source of truth for document state, versions, and ACL mappings. - **LLM gateway:** Rate limits, model fallback, prompt versioning, token accounting, and audit logs. ## Security - ACL filtering at retrieval time is mandatory (post-filtering leaks data and breaks top-k) - Per-tenant collections or payload filters, encryption at rest (KMS) and in transit (mTLS) - Temporal payload encryption via a custom data converter, since workflow history can hold content - PII redaction before embedding where required, and prompt-injection defenses (treat retrieved text as untrusted) - Audit logging of every query, retrieved chunk ID, and answer ## Scalability and Bottlenecks - **OCR and embedding throughput:** Scale workers horizontally by task queue. Use GPU workers for OCR and embeddings. - **Vector DB:** Shard by tenant, use HNSW with quantization, and keep payload indexes on filter fields. - **Reranker latency:** Cap candidates and use a small model or batch inference. - **Temporal history size:** Use child workflows and `continueAsNew` for large syncs. - **Source API rate limits:** Use a rate-limited activity queue with a token bucket. - **Cache:** Semantic and exact-match caching in Redis for repeated queries. ## Monitoring - **Temporal:** schedule-to-start latency, activity failure rate, workflow age, queue depth - **Ingestion:** freshness lag (source change to searchable), parse failure rate, chunks per doc - **Query:** p50/p95 latency per stage, cache hit rate, empty-retrieval rate - **Quality:** recall@k on a golden set, faithfulness and citation accuracy (LLM-as-judge, sampled), thumbs up/down feedback - **Cost:** tokens per query and embeddings per day - **Alerts:** DLQ growth, freshness SLA breach, error-rate spikes ## Error-Handling Summary | Failure | Strategy | |---|---| | Source API 429/5xx | Retry with backoff and jitter | | Corrupt or unsupported file | Non-retryable, mark failed, surface in admin UI | | Embedding outage | Retry, then fail over to a secondary provider | | Partial write | Saga compensation plus version pointer swap | | LLM timeout | Fallback model, or return ranked sources only | | Worker crash | Automatic replay from history | ## Recommended Improvements 1. Add a golden-set evaluation workflow in Temporal that gates model, chunking, and prompt changes. 2. Use blue/green indexes (`index_v2`) for re-embedding, with an atomic alias flip. 3. Add a knowledge-graph or metadata-aware retrieval layer for entity-heavy queries. 4. Contextual chunk headers or summaries to improve recall. 5. Per-tenant quotas and priority task queues so one large sync doesn't starve others. 6. Version prompts and retrieval configs, and log them on every request for reproducibility. 7. Define explicit SLOs, for example freshness under 15 minutes and p95 answer latency under 4 seconds.