📊 Output Requirements 1️⃣ Problem Summary Your issue is mainly coming from retrieval + chunking. The system is pulling near-matching but incorrect device instructions, leading to hallucinated hardware features. --- 2️⃣ Possible Root Causes 🔹 Chunking Chunk size too large → mixing instructions of multiple device models Overlap too small → important edge-case instructions lost Heading-based split may be inconsistent between PDF and website PDF diagrams/text not cleaned → confusing context inside chunks --- 🔹 Embeddings BGE-small-en is good but may struggle with technical hardware terms Mixed PDF + web text → inconsistent encoding quality Manual OCR text might contain noise → hurting vector similarity --- 🔹 Vector DB No metadata filter such as device model = EchoWave 300 top-k = default 5 → may return results from other models Similarity threshold not enforced → low-quality matches included --- 🔹 Query + Prompt Prompt is too short No instructions to use only retrieved context No guidance to ignore unseen device features --- 3️⃣ Practical Fixes (Simple & Actionable) ✔ Chunking Fixes Reduce chunk size to 250–300 tokens Increase overlap to 20–25% for preserving instructions Use semantic chunking (by sections like reset, troubleshoot, safety) Clean OCR text (remove garbage characters, merge broken lines) --- ✔ Embedding Fixes Switch to bge-base-en-v1.5 (better recall for technical content) Re-embed PDFs using cleaned text Tag data with metadata: device_model, version, type --- ✔ Vector DB Fixes Add metadata filter: device_model="EchoWave 300" Increase top-k from 5 → 8–10 Enforce minimum similarity threshold (e.g., ≥ 0.25) --- ✔ Query + Prompt Fixes Use a stronger prompt like: > “Use ONLY the retrieved context. If the answer cannot be found, say ‘Not available in the documentation.’ Do NOT hallucinate.” Add citation requirement Add instruction: “If multiple models appear, choose only EchoWave 300.” --- 4️⃣ Testing Plan A/B chunking: 500 vs 300 tokens A/B embeddings: bge-small vs bge-base top-k testing: 5 vs 10 vs 15 Prompt variations: short vs constrained vs citation-based Try reranking: Cohere reranker or Voyage ranker for top-k reduction --- 5️⃣ Hallucination Reduction Tips Always force source-only answers Add fallback: “No information found in the docs.” Ask the model to list the exact chunks used Force it to show confidence levels Add binary step: “Check if context actually contains answer = yes/no” --- 6️⃣ Final Recommendation Most likely cause: Wrong device model chunks retrieved due to mixed embedding quality + missing metadata Suggested fix: Add metadata filters + reduce chunk size + improve prompt constraints Next steps: Re-embed after cleaning PDFs → set metadata → test new chunking → enable reranker ---