# RAG Architecture for Accuracy-First AI Assistant ## What You're Actually Building You need a medium-scale AI assistant that prioritizes getting answers *right* over getting them *fast* or *cheap*. With 1K-10K users and up to 5K documents, you're past the "just throw it in a vector DB" stage but not yet at enterprise complexity. Your $50-300/month budget is tight enough that you can't waste money on oversized infrastructure, but generous enough to invest in quality where it matters. **The core tension:** Accuracy usually costs money (better embeddings, reranking, larger context windows) but your budget caps how much you can spend. The solution is to be surgical—spend on components that directly improve retrieval quality, cut corners on infrastructure that just needs to work. **Monthly cost estimate:** ~$180-240 **Typical query latency:** 800-1200ms (you're choosing accuracy over speed) **Complexity:** Medium (some moving parts, but nothing exotic) --- ## The Retrieval Quality Stack ### Embeddings: Where Accuracy Starts **Use `text-embedding-3-large` from OpenAI** (3072 dimensions) Why this specific choice: - It's OpenAI's most accurate embedding model, and for 500-5K documents, you'll embed maybe 100K-500K chunks total (one-time cost: $13-65) - Ongoing query costs are negligible at medium scale (~10K queries/day = $3-4/month) - The dimension count matters here: 3072 dims capture semantic nuance that 1536-dim models miss, which directly impacts whether you retrieve the right context Don't use `text-embedding-3-small` even though it's 5x cheaper—you said accuracy is priority, and this is literally the foundation of your entire system. **Cost reality:** ~$4-6/month for queries + one-time ingestion cost --- ### Vector Storage: Pinecone Serverless **Go with Pinecone's serverless tier**, not self-hosted Qdrant or Weaviate. Here's why this fits: - At 500-5K docs (likely 5K-50K chunks), you're in the sweet spot where Pinecone serverless is cheap ($20-40/month) but still performant - You don't have time to babysit a self-hosted database—serverless means no server management, automatic scaling, and 99.9% uptime - Pinecone's HNSW implementation is battle-tested for accuracy (ef_construction=200 gives you excellent recall) **Index configuration:** - Metric: cosine similarity (standard for OpenAI embeddings) - Pods: serverless (you pay per query, not per hour) - Metadata: store document_id, chunk_position, section_title, timestamp **When this breaks:** If you hit 100K+ chunks or need <100ms retrieval, you'd migrate to Pinecone's pod-based tier (~$70/month) or consider self-hosted. But at medium scale, serverless is the correct choice. **Cost:** ~$25-45/month depending on query volume --- ### Chunking: Semantic Boundaries Matter Don't just split on 512 tokens blindly. For an AI assistant, your chunks need to be self-contained enough to make sense alone. **Strategy:** - Base chunk size: 800 tokens (larger than typical because you're prioritizing accuracy—more context per chunk reduces "partial information" retrievals) - Overlap: 200 tokens (25% overlap ensures you don't split important context across boundaries) - Recursive splitting: try to break on paragraph boundaries first, then sentences, then tokens **Handle special content:** - Tables: keep entire table in one chunk if <800 tokens, otherwise split by rows but include headers in each chunk - Code blocks: never split mid-function; treat each function/class as atomic - Lists: keep list items with their parent context **The mistake everyone makes:** Using tiny chunks (256-512 tokens) because they read it's "best practice." Small chunks give you *precision* but hurt *recall* and context coherence. At 800 tokens, you capture enough semantic meaning that each chunk can stand alone. --- ### Retrieval: Hybrid + Reranking This is where you spend your accuracy budget. **Pipeline:** ``` Query → Semantic Search (k=20) → Keyword Filter → Rerank (top 5) → Context ``` **Why hybrid matters here:** 1. **Semantic search** (k=20): Cast a wide net with vector similarity. You're retrieving 20 candidates because you'll filter them down, and you want high recall at this stage. 2. **Keyword filter:** If the query contains specific technical terms, entity names, or dates, add a metadata filter or BM25 sparse retrieval. This catches exact matches that embeddings might miss (e.g., "What did the Q3 2024 report say about revenue?" needs to filter on "Q3 2024"). 3. **Reranking with Cohere Rerank 3:** This is your accuracy multiplier. Take those 20 candidates and rerank them with a cross-encoder model that actually understands query-document relevance better than embeddings alone. Keep the top 5 for context. **Cost breakdown:** - Cohere Rerank: $2 per 1000 searches (if 10K queries/day, that's ~$600/month—too expensive) - **Smarter approach:** Only rerank when confidence is low or query is complex. Use a simple heuristic: if top-3 semantic results have similarity <0.75, trigger reranking. This cuts rerank calls by 60-70%. - Realistic rerank cost: ~$50-80/month **Latency:** - Embedding query: ~50ms - Pinecone search: ~100-150ms - Reranking (when used): ~400-600ms - Total: 550-800ms (without rerank), 1000-1400ms (with rerank) You're choosing accuracy, so 1 second latency is acceptable. --- ### Context Assembly: Token Budget Discipline Assume you're using GPT-4 Turbo (128K context) or Claude Sonnet (200K context). Don't let "large context" make you sloppy. **Token allocation for each request:** - System prompt + instructions: 500 tokens - Retrieved context: 4000 tokens (5 chunks × 800 tokens each) - Conversation history: 1500 tokens (last 3-4 turns) - User query: 300 tokens - Response budget: 1500 tokens - **Total input:** ~6300 tokens (well within limits) **When retrieved content exceeds 4000 tokens:** 1. Score each chunk by reranker score + recency + source authority 2. Keep highest-scoring chunks up to budget 3. If chunks are from same document, include a one-sentence summary of excluded sections **Example:** If you retrieve 5 chunks but they total 5000 tokens, drop the lowest-scored chunk and add a line: "Additional related information was found in [Document X, Section Y] but omitted for brevity." --- ## Getting This Live in 3 Weeks **Week 1: Core Pipeline** Day 1-2: Document ingestion - Write a script that reads your docs, chunks them (800/200), and generates embeddings - Store chunks in Pinecone with metadata (source, section, timestamp) - Test with 50-100 real documents Day 3-4: Basic retrieval - Build semantic search: query → embed → Pinecone search (k=20) - No reranking yet, just return top 5 by similarity - Create 20 test queries covering common user needs Day 5: Initial testing - Run your 20 queries, manually score relevance (are the right chunks in top 5?) - Target: >60% of queries should have at least 1 highly relevant chunk in top 5 **Week 2: Accuracy Improvements** Day 1-3: Add reranking - Integrate Cohere Rerank 3 - Compare results with and without reranking on your test set - Tune the confidence threshold for when to trigger reranking Day 4-5: Metadata filtering - Add keyword matching for entity names, dates, technical terms - Test queries like "Q3 2024 revenue" or "John Smith's proposal" - Measure improvement on exact-match queries **Week 3: Production Ready** Day 1-2: Cost monitoring - Log every API call (embeddings, Pinecone, reranking, LLM) - Calculate actual cost per 1K queries - If over budget, adjust reranking trigger threshold Day 3-4: Evaluation harness - Build a test set of 100 queries with labeled "good" and "bad" retrievals - Automate scoring: precision@5, recall@5, MRR - Set up weekly automated runs to catch regression Day 5: Production deployment - Add error handling (what if Pinecone is down? Fall back to direct LLM) - Set up basic alerting (latency >2s, error rate >5%) - Soft launch with 10% of users --- ## Know Before You Launch **Evaluation targets:** | Metric | Target | How to Measure | |--------|--------|----------------| | Precision@5 | >80% | At least 4/5 retrieved chunks are relevant | | User satisfaction | >75% thumbs up | Track explicit feedback on answers | | P95 latency | <1500ms | Log all query times, check 95th percentile | | Cost per 1K queries | <$4 | Embeddings + Pinecone + Rerank + LLM | **Create a labeled test set:** - 100 real user queries (or realistic examples) - For each query, manually mark which chunks *should* be retrieved - Run your pipeline weekly and measure precision/recall **Use LLM-as-judge for answer quality:** - Give GPT-4 the query, retrieved context, and generated answer - Ask it to score accuracy (1-5) and completeness (1-5) - Track median scores over time --- ## What Could Go Wrong | Risk | Likelihood | Impact | Mitigation | |------|------------|--------|------------| | Reranking costs blow budget | Medium | High | Set hard limit on rerank calls per day, use confidence-based trigger | | Poor retrieval on edge-case queries | High | Medium | Build feedback loop where users flag bad answers, add to test set | | Document updates break existing chunks | Medium | Medium | Version your embeddings, maintain changelog, re-embed only changed sections | | Pinecone serverless has cold start lag | Low | Medium | Keep one "heartbeat" query every 5 minutes to avoid cold starts | **Biggest unknown:** You won't know your *actual* accuracy until real users try it. Your test queries are educated guesses. Plan to iterate heavily in weeks 4-6 based on user feedback. --- ## The Money Reality | Component | Monthly Cost | Assumptions | |-----------|--------------|-------------| | OpenAI embeddings | $5 | 10K queries/day × 30 days × $0.13/1M tokens | | Pinecone serverless | $35 | ~20K chunks, 10K queries/day | | Cohere Rerank | $60 | 30% of queries trigger reranking | | GPT-4 Turbo calls | $120 | 10K queries/day × 6K input + 1.5K output tokens | | **Total** | **$220** | | This fits your $50-300 budget at the higher end. If you need to cut costs: **Drop to $150/month:** - Use Claude Sonnet instead of GPT-4 Turbo (saves ~$80/month) - Reduce reranking to 15% of queries (saves ~$40/month) - Accept slightly lower accuracy **Drop to $100/month:** - Switch to `text-embedding-3-small` (saves negligible $2/month, hurts accuracy 5-10%) - Use only semantic search, no reranking (saves $60/month, hurts accuracy 15-20%) - Self-host Qdrant on a $20/month VPS (saves $15/month, adds DevOps burden) **My recommendation:** Start at $220/month. You said accuracy is priority, and you have the budget. If users love it, the cost is justified. If they don't, you have bigger problems than $70/month in savings. --- ## One Final Thing Your biggest risk isn't technical—it's that you spend 3 weeks building this and then discover your users' actual questions don't match your test set. Before you write a single line of code: 1. Collect 50 real queries from users (or simulate them) 2. Manually try to answer them using your documents 3. Note which questions are *actually hard* vs which are simple lookups If 80% of queries are simple ("What's the return policy?"), you're over-engineering. If 80% are complex ("Compare the benefits of Plan A vs Plan B for a family of 4 with pre-existing conditions"), you need this architecture—and maybe more. Build for the 80%, not the 20%.