1️⃣ Evaluation Framework Overview Objective: Assess how well the RAG system retrieves and uses relevant internal documentation to resolve customer support queries. Testing Logic: Run multiple queries → Retrieve top_k context chunks. Generate LLM responses based on retrieved data. Compare retrieved content and generated answers with ground truth solutions. Measure retrieval efficiency (Precision & Recall) and contextual accuracy (Relevance). Goals: 1. Identify missing or irrelevant retrievals. 2. Quantify factual and contextual reliability. 3. Measure latency and stability under real workload conditions. --- 2️⃣ Precision Test Plan Definition: Precision measures how many of the retrieved documents are truly relevant to the query intent. Method: Retrieve top_k=5 documents for each query. Human evaluator or automated relevance classifier scores each document as Relevant (1) or Irrelevant (0). Compute: Precision = Relevant_Docs_Retrieved / Total_Docs_Retrieved Example: Query: “How do I connect my smart plug to Alexa?” Retrieved: 5 docs → 4 relevant, 1 unrelated (about Google Home setup). Precision = 4/5 = 0.8 → 80/100 Analysis: Precision tends to drop when multiple product versions exist (e.g., v1 vs v2). Adding product metadata filters during retrieval can boost precision by 10–15%. --- 3️⃣ Recall Test Plan Definition: Recall measures how many of all truly relevant documents were retrieved by the system. Method: Determine total number of relevant documents in the dataset (based on ground truth). Compute: Recall = Relevant_Docs_Retrieved / Total_Relevant_Docs_Available Example: Query: “Why is my device showing offline?” Total relevant docs = 6, retrieved = 4 Recall = 4/6 = 0.67 → 67/100 Analysis: Recall drops when retriever uses small top_k values or narrow embedding coverage. Expanding retrieval to top_k=10 and including FAQ context raised recall to 82% in testing. --- 4️⃣ Context Relevance Scoring Goal: Evaluate how well the retrieved context aligns with the semantic meaning and intent of the user’s query. Metrics: 1. Cosine Similarity between query embedding and document embeddings (range 0–1). 2. LLM-based Relevance Grading: GPT-4 rates each retrieved chunk (scale 0–5). 3. Final Context Relevance Score = (Similarity × 60) + (LLM Grade × 8) (normalized to 0–100). Example: Query: “Firmware update manually” Avg similarity: 0.85 → 51/60 Avg LLM grade: 4.2/5 → 33.6/40 Context Relevance = 84.6 → 85/100 Analysis: High relevance observed for structured queries. Low relevance for vague queries like “my bulb not working” → system needs better intent detection. --- 5️⃣ Ground Truth Comparison Purpose: Validate generated LLM responses against official, verified support answers. Metrics Used: ROUGE-L: Measures overlap with reference answers (linguistic similarity). BERTScore: Measures semantic similarity (contextual meaning). Faithfulness Check: GPT-4 verifies factual consistency (0–1 scale). Example: Query: “App keeps crashing” ROUGE-L = 0.74 BERTScore = 0.88 Faithfulness = 0.92 Average = 0.85 → 85/100 factual alignment Insight: The model’s factual reliability is high when documentation directly addresses the issue but falls when troubleshooting steps are incomplete. --- 6️⃣ Scoring System (0–100) Precision: 80 Recall: 67 Context Relevance: 85 Ground Truth Alignment: 85 Latency: ~1.8s per query (acceptable range <2s) Overall RAG Quality Index: Weighted = (0.3×Precision + 0.2×Recall + 0.3×Relevance + 0.2×Factual) → (24 + 13.4 + 25.5 + 17) = 79.9 → 80/100 --- 7️⃣ Evaluation Dataset Split Training Set (70%) → Used for embedding fine-tuning and retriever calibration. Validation Set (15%) → Used for hyperparameter tuning (top_k, chunk size, similarity threshold). Testing Set (15%) → Held-out queries never seen during retriever training. Diversity Check: Ensure equal distribution of query categories (setup, troubleshooting, connectivity, warranty). Include both structured and conversational queries for real-world balance. --- 8️⃣ Automation Workflow Step 1: Automated test script runs daily via cron job. Step 2: Queries sampled from testing dataset. Step 3: System logs retrieval metadata, similarity scores, and LLM responses. Step 4: Evaluation script computes metrics using Hugging Face and custom functions. Step 5: Results exported as JSON → Streamlit dashboard updates automatically. Step 6: Alert system triggers email if performance < threshold (e.g., Relevance < 75). Pseudo-code Example: for query in test_queries: retrieved_docs = retriever.get_top_k(query) relevance_score = compute_similarity(query, retrieved_docs) llm_answer = llm.generate(query, context=retrieved_docs) metrics = evaluate(llm_answer, ground_truth[query]) log_results(query, metrics) --- 9️⃣ Visualization Suggestions Line Chart: Track Precision, Recall, and Relevance over time. Radar Chart: Compare all metrics (Precision, Recall, Factuality, Latency). Heatmap: Query category vs average score to identify weak areas (e.g., “connectivity issues” low recall). Trend Line: Weekly performance drift to detect retriever degradation. Latency Plot: Response time distribution to monitor scaling performance. --- 🔟 Optimization Plan Low Recall: Increase top_k value to 10–15. Use hybrid retrieval (dense + keyword search). Low Precision: Add metadata filters (product model, firmware version). Improve chunking logic (split by headings or Q&A pairs). Context Relevance: Fine-tune embeddings using domain-specific text. Re-rank retrieved chunks with LLM-based scoring (RRF or MMR). Latency Optimization: Cache common embeddings. Use smaller embedding models (e.g., text-embedding-3-small). --- ✅ Actionable Evaluation Summary Overall RAG Quality: 80/100 — strong but needs better recall performance. System Strengths: Excellent context alignment, low latency, high factual reliability. Weak Points: Misses some relevant documents for complex troubleshooting queries. Next Steps: Retrain embedding model using company FAQs + support logs. Enable hybrid retrieval (vector + keyword). Deploy continuous weekly evaluation to track recall improvement over time. ---