### Phase 1 — 🏗️ Retrieval System Reconstruction ### 1.1 Current pipeline map 1. User Query Original text and constraints 2. Query Processing Normalization and intent analysis — unknown implementation 3. Parallel Candidate Retrieval Dense top-10 + BM25 top-10 4. Fusion Combines candidate lists — method unknown 5. Reranking Top-5 candidates selected for reranking or output — exact behavior to verify 6. Final Evidence Selection Context assembly and source traceability — unknown 7. Downstream RAG Answer Uses retrieved evidence to generate an answer Important: The diagram represents the proposed reconstruction based on the sample inputs, not a verified implementation. The exact location of reranking, context assembly, filtering and error handling must be confirmed from code or logs. ### 1.2 Component inventory | Component | Current evidence | Main risk | | ----------------- | ---------------------------- | ---------------------------------------------- | | PDF ingestion | 10,000 PDFs reported | Missing or stale files | | Parsing | Unknown | Lost tables, headings or OCR errors | | Chunking | 500 tokens, 50 overlap | Cross-section facts may be separated | | Embeddings | Model unknown | Semantic mismatch cannot be assessed | | Vector index | Present by input description | Index coverage and freshness unknown | | BM25 | Present by input description | Exact identifiers may be poorly normalized | | Fusion | Present, method unknown | One branch may suppress useful candidates | | Reranker | Cross-encoder reported | Relevant candidates may be excluded beforehand | | Context selection | Unknown | Required evidence may be dropped | | Evaluation | No labeled benchmark | Improvements cannot yet be demonstrated | ### 1.3 Configuration inventory * Dense retrieval: top-10. * BM25 retrieval: top-10. * Fused candidates: top-10. * Reranker candidates: top-5. * Chunk size: 500 tokens. * Chunk overlap: 50 tokens. * Embedding model, fusion algorithm, metadata filters, reranker model and latency limits: unknown. ### 1.4 Evidence gaps * Source-to-index reconciliation report. * Query-level results from each retrieval branch. * Raw scores, ranks and fusion details. * Chunk text, source IDs and document versions. * Reranker input/output logs. * Final context-selection logs. * Ground-truth relevance judgments. * Latency and cost measurements. ### 1.5 Initial risk register | Risk | Priority | Reason | | --------------------------------- | -------- | ------------------------------------------------------------- | | Exact policy IDs missed | High | Important identifiers need reliable exact matching | | Multi-section evidence incomplete | High | A correct individual chunk may not contain all required facts | | Relevant results ranked too low | High | Candidate generation and ranking need separation | | PDF extraction failures | Medium | Possible but not yet evidenced | | Excessive retrieval latency | Unknown | No measurements supplied | Phase 1 result: The current system is partially reconstructed. Its actual root causes remain unverified. ### Phase 2 — 🩺 Retrieval Failure Diagnosis Engine ### 2.1 Failure register F-01 Exact policy ID missed * Symptom: Queries containing policy identifiers fail to retrieve the expected document. * Expected evidence: The authoritative policy containing the exact ID. * Actual evidence: Not supplied; only the failure pattern is known. * Possible stages: Corpus, parsing, indexing, lexical retrieval, filtering or fusion. * Hypotheses: Identifier absent from corpus; punctuation or OCR corruption; identifier indexed in a non-searchable field; restrictive filter; low candidate depth. * Verification: Search the original source, extracted text, index records and raw BM25 results for the exact identifier. * Remediation: Fix the verified failure, then test exact-match retrieval with punctuation and spacing variants. * Confidence: Low for any specific root cause. F-02 Relevant evidence appears at low ranks * Symptom: The correct document exists in returned results but ranks too low. * Possible stages: Candidate generation, score fusion, reranking or final selection. * Hypotheses: Weak lexical matching, semantic mismatch, inappropriate fusion, poor reranker relevance or duplicate crowding. * Verification: Track the same relevant document through dense results, BM25 results, fused results, reranked results and final context. * Remediation: Change only the stage where its rank deteriorates. * Confidence: Medium that ranking is involved; low for the specific cause. F-03 Multi-section evidence is incomplete * Symptom: A question requiring two sections retrieves only one necessary piece of evidence. * Possible stages: Query decomposition, chunk boundaries, candidate depth, ranking or context selection. * Hypotheses: One query retrieves only one section; the second section ranks too low; headings and dependencies are separated; the context limit removes a needed chunk. * Verification: Label both required evidence units and trace each through the entire pipeline. * Remediation: Test query decomposition, parent-child retrieval or structure-aware chunking as separate experiments. * Confidence: Medium that evidence coverage is the issue; root cause unknown. ### 2.2 Distinguishing the failure types | Diagnostic result | Likely classification | | ---------------------------------------------------------- | --------------------------- | | Original document does not contain the information | Corpus absence | | Source contains information, extracted text does not | Parsing/extraction failure | | Extracted text contains information, index record does not | Indexing failure | | Index contains it, neither search branch retrieves it | Candidate retrieval failure | | One branch retrieves it, fusion drops it | Fusion failure | | Fused candidates contain it, reranker demotes it | Reranking failure | | Ranked results contain it, final context omits it | Context-selection failure | Required instrumentation: Log a query ID, original and transformed queries, applied filters, candidate IDs, branch ranks and scores, fused rank, reranker rank, final selected chunks, source versions, errors and stage latency. Do not change embeddings or chunk sizes until these checks identify a plausible failure mechanism. ### Phase 3 — 🧭 Query Analysis and Transformation The first goal is to preserve the original question while creating additional retrieval paths only when justified. | Query pattern | Proposed action | Example | | ----------------------------- | --------------------------------------------------------------------------- | ----------------------------------------- | | Exact policy ID | Preserve original query and perform exact identifier lookup | POL-4821 | | Acronym or domain synonym | Add a validated synonym variant | MFA and multi-factor authentication | | Conceptual paraphrase | Compare original-query dense search with a carefully normalized version | “How do I regain account access?” | | Two independent requirements | Decompose into evidence subqueries | “Find eligibility and required documents” | | Time-specific policy question | Preserve date constraints and filter version/effective date where supported | “Policy effective in June 2026” | | Negation or exception | Preserve negation; do not rewrite away the exception | “Which users are not eligible?” | | Ambiguous term | Retrieve with original wording and compare candidate interpretations | “Access control” | ### Transformation experiment example Original query: “What is the maximum reimbursement under POL-4821, and which receipts are required?” * Query A: Original query unchanged. * Query B: Exact identifier lookup for `POL-4821`. * Query C: Two subqueries: maximum reimbursement and required receipts, each constrained to the same policy. * Preserve: policy ID, maximum amount, receipts, and the fact that both answers are required. * Drift risk: A rewrite may retrieve another policy or omit the receipts requirement. * Evaluation: Recall of both required evidence units, identifier precision, nDCG@K and latency. * Acceptance: Both required evidence units are retrieved without increasing unauthorized or wrong-policy results. HyDE should not be the first intervention for exact identifiers. It is a candidate experiment for conceptual queries only if baseline failures support it. ### Phase 4 — 🧩 Chunking Strategy Evaluation ### 4.1 Comparison matrix | Strategy | Strength | Failure risk | Suitable content | Evaluation needed | | --------------------- | ------------------------------------------ | ------------------------------------- | --------------------------------- | ----------------------------- | | Fixed-size | Simple, predictable | Splits headings and procedures | Uniform prose | Baseline | | Recursive token-based | Respects preferred boundaries | May still split semantic dependencies | General PDFs | Compare evidence coverage | | Structure-aware | Preserves headings and sections | Depends on parsing quality | Manuals and policies | Heading and section retrieval | | Semantic chunking | Groups related concepts | Variable size and extra processing | Mixed-topic documents | Recall, latency, index size | | Parent-child | Small precise matches with broader context | Larger context and duplicate overlap | Policies and long manuals | Completeness and precision | | Table-aware | Preserves row/column relationships | Incorrect extraction of merged cells | Pricing, limits and policy tables | Cell-level evidence accuracy | | Code-aware | Preserves syntax and logical units | Language-specific segmentation errors | Code and technical manuals | Function-level retrieval | ### 4.2 Initial recommendation Keep the current 500/50 configuration as the baseline. Test structure-aware chunking first because the corpus contains manuals and policies, and the reported failure involves evidence across sections. Next, test parent-child retrieval for multi-section questions. Use table-aware parsing for documents containing policy tables. These are experiments, not confirmed fixes. ### 4.3 Controlled experiment plan * Baseline: Existing 500-token chunks with 50-token overlap. * Treatment A: Structure-aware section chunks. * Treatment B: Structure-aware child chunks with parent-section expansion. * Treatment C: Table-aware extraction on a table-containing subset. * Hold constant: Corpus version, embedding model, retrieval filters, query set, fusion and ranking configuration. * Measure: Recall@10, nDCG@10, evidence-unit coverage, exact ID success, index size, indexing time and query latency. * Acceptance: Improve the preselected target metric beyond a predefined threshold, with no unacceptable regression in other query groups. * Regression test: Re-run all exact-ID, table, multi-section and no-answer cases. The threshold must be set before running the experiment; no benchmark result is available yet. ### Phase 5 — ⚖️ Hybrid Search Configuration Designer ### 5.1 Proposed provisional configuration | Component | Proposed configuration | Rationale | Trade-off | Validation | | --------------- | ------------------------------------- | ----------------------------------------------- | -------------------------------------- | ------------------------------ | | Dense retrieval | Top-20 candidates | Test improved semantic candidate recall | More compute | Recall and latency sweep | | BM25 | Top-20 candidates | Improve keyword and identifier recall | More candidates and duplicates | Exact-ID success | | Exact ID branch | Dedicated exact lookup, if supported | Protect policy identifiers | Requires normalized ID field | ID retrieval precision | | Fusion | Reciprocal Rank Fusion (RRF) baseline | Avoid comparing incompatible raw scores | Loses some score magnitude information | Compare against current fusion | | Fused pool | Top-20 | Retain more candidates for reranking | Higher reranking cost | Recall and latency sweep | | Reranking | Up to 20 candidates | Test whether relevant results move up | Additional inference latency | nDCG and MRR | | Final evidence | Up to 5 chunks initially | Limit context volume while testing completeness | May omit a second evidence unit | Evidence coverage | All proposed candidate counts are provisional test values, not optimized settings. ### 5.2 Filter order * Authenticate the requester and establish tenant and document permissions. * Enforce authorization constraints in every search branch. * Retrieve eligible candidates. * Fuse candidate rankings. * Deduplicate while preserving source identity. * Rerank eligible candidates. * Select evidence based on relevance, authority, freshness and completeness. * Record source provenance and final context. If the current database cannot enforce a required filter, the team must establish a secure equivalent before release. ### Phase 6 — 📈 Candidate Retrieval and Top-K Optimization Test retrieval depth independently of chunking and query rewriting. | Configuration | Dense K | BM25 K | Fused K | Reranker K | Final chunks | | ---------------------- | ------- | ------ | ------- | ---------- | ------------ | | C0 — Current | 10 | 10 | 10 | 5 | 5 | | C1 — Wider candidates | 20 | 20 | 20 | 20 | 5 | | C2 — Broad retrieval | 50 | 50 | 40 | 20 | 5 | | C3 — Precision control | 20 | 20 | 20 | 10 | 5 | Final chunk counts are proposed caps, not evidence that five chunks will always be sufficient. Procedure: * Freeze a labeled query set and corpus version. * Run each configuration on the same queries. * Record recall before fusion, after fusion, after reranking and after final selection. * Record p50/p95 latency and per-query cost. * Report results for exact IDs, conceptual queries, multi-section questions and no-answer cases separately. * Select the least expensive configuration that meets predefined quality and security thresholds. A larger candidate pool is justified only if it recovers relevant evidence at an acceptable cost. ### Phase 7 — 🏆 Ranking and Reranking Engineering ### Ranking diagnosis * If the correct document never enters the fused candidate pool, improve candidate generation. * If it enters a branch but disappears after fusion, investigate fusion and candidate truncation. * If it reaches the reranker but drops in rank, inspect the reranker model, input truncation and relevance labels. * If it remains highly ranked but disappears from the final context, investigate selection limits and deduplication. ### Proposed reranking policy * Rerank only authorized, deduplicated candidates. * Include enough query and document text for the relevance decision. * Use relevance as the principal ranking signal. * Use freshness and source authority as controlled tie-breakers where appropriate. * Preserve different evidence units needed to answer multi-part questions. * Keep exact identifiers and document versions in the reranker input. * On reranker failure, use the validated fused ranking as a degraded fallback and log the error. * Never silently replace an authoritative policy with a lower-authority duplicate. Experiment: Compare the current reranker with no reranking and the proposed wider reranking pool. Acceptance: Better nDCG@10 or MRR on the target query segments, no unacceptable regression on exact identifiers, and acceptable latency. ### Phase 8 — 🗂️ Indexing, Metadata and Corpus Quality ### Required audit checks | Check | Method | Failure indicated | | ------------------------ | -------------------------------------------------- | ----------------------------------- | | Source-to-index coverage | Compare source manifest with indexed document IDs | Missing indexed documents | | Extraction quality | Compare sampled PDF text with source pages | OCR or parsing loss | | Chunk completeness | Inspect chunk boundaries against source sections | Omitted dependencies | | Metadata completeness | Validate required fields | Missing filter/provenance values | | Freshness | Compare source versions and index timestamps | Stale evidence | | Identifier integrity | Compare source IDs with extracted and indexed IDs | OCR or normalization errors | | Duplicate detection | Compare hashes and near-duplicate text | Candidate crowding | | Authorization | Test access using different permitted roles | Potential leakage or over-filtering | | Deletion handling | Confirm deleted/superseded records are invalidated | Outdated policies remain searchable | ### Reconciliation workflow Source manifest → extraction verification → chunk manifest → index manifest → metadata validation → discrepancy report → controlled reindexing → post-index validation. Do not fix retrieval recall by removing authorization filters. Correct the underlying indexing or permission defect. ### Phase 9 — 🧮 Retrieval Evaluation Framework Build a benchmark before comparing retrieval strategies. ### Metrics | Metric | Purpose | Limitation | | ------------------- | -------------------------------------------------------- | ---------------------------------------------- | | Recall@K | Measures relevant evidence recovered | Does not measure ranking order | | Precision@K | Measures proportion of retrieved results judged relevant | Can penalize useful complementary evidence | | MRR | Rewards placing the first relevant result high | Misses completeness of multiple evidence units | | nDCG@K | Evaluates ranking with graded relevance | Depends on consistent judgments | | Hit Rate@K | Measures whether any relevant result appears | Weak for multi-part questions | | Evidence coverage | Measures required evidence units recovered | Requires annotated evidence units | | Exact-ID success | Tests precise identifier retrieval | Needs exact identifier ground truth | | p50/p95/p99 latency | Measures operational response time | Requires representative workload | | Cost per query | Measures operating expense | Depends on accurate cost accounting | ### Benchmark construction Start with a proposed pilot of 100–200 queries, then expand based on corpus size, query diversity and observed failure patterns. This is a planning target, not an existing dataset. Include: * Common policy questions. * Exact policy ID lookups. * Rare terms and acronyms. * Paraphrases. * Multi-section questions. * Tables and numerical limits. * Long-document questions. * Temporally constrained queries. * Conflicting or superseded policies. * No-answer queries. * Permission-restricted documents. For each query, label the required evidence units, acceptable source documents, authority/version constraints, expected access permissions and whether the corpus contains an answer. Have domain reviewers adjudicate ambiguous cases. Quantitative comparisons remain pending until labels exist. ### Phase 10 — 🧪 Ablation and Controlled Experiments | ID | Hypothesis | Baseline | Change | Metrics | Acceptance criteria | | ---- | ----------------------------------------------- | ----------------- | --------------------------------- | -------------------------- | ------------------------------------------------------ | | E-01 | BM25 adds exact-ID recall | Dense only | Add BM25 | Exact-ID success, Recall@K | Target improvement without unacceptable precision loss | | E-02 | Fusion improves combined retrieval | Current fusion | RRF | Recall, nDCG, latency | Better target quality at acceptable cost | | E-03 | Structure-aware chunks preserve section context | Fixed chunks | Structure-aware chunks | Evidence coverage, Recall | Better completeness without unacceptable regression | | E-04 | Wider pools recover missing candidates | Current K | Larger K values | Recall, latency, cost | Best quality-cost trade-off | | E-05 | Reranking improves ordering | Fused ranking | Cross-encoder reranking | MRR, nDCG | Improvement exceeds predefined threshold | | E-06 | Query decomposition helps multi-part queries | Original query | Targeted subqueries | Evidence coverage, latency | Both required evidence units recovered more often | | E-07 | Deduplication increases diversity | Current selection | Deduplicate near-identical chunks | Precision, coverage | Less redundancy without losing unique evidence | For every experiment, freeze the dataset and corpus, log software/model versions, run the same queries and preserve individual regressions. Where results are close, use paired statistical analysis and sufficient repeated runs. Do not claim statistical significance without evidence. ### Phase 11 — 🧠 Retrieval Failure Pattern Library | Pattern | Discriminating test | Possible intervention | Main trade-off | | -------------------------------------------- | ---------------------------------------------- | ------------------------------------------ | --------------------------------- | | Relevant evidence absent from candidates | Inspect both branch outputs | Fix corpus/index or candidate generation | More indexing/search work | | Correct evidence ranks low | Trace rank changes by stage | Tune fusion or reranking | More latency | | Keyword beats semantic search | Segment by exact-term queries | Improve lexical matching | Less semantic generalization | | Semantic beats lexical search | Test paraphrases | Improve dense retrieval | Potential exact-term misses | | Hybrid worsens relevance | Compare branch and fused rankings | Change fusion or routing | Added complexity | | Chunk splits related facts | Inspect source boundaries | Structure-aware or parent-child retrieval | Larger context | | Large chunks add noise | Inspect irrelevant text per chunk | Smaller structured children | More index entries | | Filters exclude relevant records | Compare authorized and filter-eligible records | Correct metadata/filter logic | Must preserve security | | Duplicates crowd out evidence | Measure duplicate concentration | Deduplicate | Risk of merging distinct versions | | Stale evidence is retrieved | Compare indexed/source versions | Reconcile index freshness | Reindexing cost | | Reranker harms a query segment | Segment-level before/after ranking | Change model/policy or disable selectively | Quality variability | | High recall, low precision | Inspect irrelevant candidate rate | Improve fusion/reranking | Possible recall reduction | | Evidence incomplete | Check required evidence units | Decomposition or context expansion | More latency/tokens | | No-answer query retrieves misleading content | Test known unanswerable questions | Evidence sufficiency checks | More abstentions | These are diagnostic hypotheses, not proof of a root cause. ### Phase 12 — ⚡ Latency, Cost and Scalability ### Measurement plan Record per stage: * Query preprocessing. * Query embedding. * Dense search. * BM25 search. * Fusion and deduplication. * Reranking. * Context assembly. * End-to-end latency. Report p50, p95 and p99 for representative traffic. Measure embedding/indexing costs separately from per-query costs. | Optimization | Expected quality effect | Operational trade-off | Verification | | ------------------------- | --------------------------------------- | -------------------------- | -------------------------------------- | | Selective reranking | Can retain quality on difficult queries | Routing complexity | Compare routed vs all-query reranking | | Candidate pruning | May reduce later-stage cost | Can lose relevant evidence | Recall before and after pruning | | Cache normalized queries | Can reduce repeated work | Staleness and access risks | Cache hit rate and authorization tests | | Reduce redundant branches | May lower latency | Potential recall loss | Branch ablation | | Batch reranking | May improve throughput | Queueing delay | Throughput and p95 latency | | Optimize index updates | May improve freshness | Operational complexity | Source-to-index delay | No latency or cost estimates can be responsibly supplied from the inputs alone. ### Phase 13 — 🔄 Fallback and Degraded-Mode Design | Failure | Fallback | Safeguard | | ------------------------ | ------------------------------------------------------------------ | --------------------------------------------- | | Vector search fails | Attempt BM25 if healthy | Preserve access filters | | BM25 fails | Attempt dense search if healthy | Flag degraded retrieval | | Reranker fails | Use validated fused order | Log failure; do not claim reranking succeeded | | Empty results | Verify filters, corpus coverage and backend health | Distinguish no answer from search failure | | Partial results | Return validated available evidence with an incomplete-status flag | Do not imply completeness | | Timeout | Apply bounded retry only for transient errors | Respect deadline and retry budget | | Index stale | Use only a permitted, validated index state | Flag freshness uncertainty | | Permission service fails | Fail closed where authorization cannot be established | Never bypass access controls | Timeouts and retry counts must be set from service-level objectives and backend capabilities. A backend failure must never be silently interpreted as proof that the corpus contains no answer. ### Phase 14 — 🛡️ Security and Evidence Integrity ### Mandatory controls * Enforce tenant and document-level permissions during retrieval. * Apply the same authorization constraints to dense, lexical, exact-match and fallback paths. * Partition or validate caches by the appropriate authorization scope. * Invalidate or revalidate permissions after access changes. * Treat retrieved documents as untrusted content, not executable instructions. * Preserve source authority, version, effective date and supersession status. * Detect conflicting sources and retain both when needed for review. * Record provenance for every final evidence chunk. Minimum evidence record: * Query ID. * Source document ID and version. * Chunk ID. * Relevant metadata and permission decision. * Retrieval branch and original rank. * Fusion and reranker ranks. * Score provenance, where available. * Final selection status. A highly relevant document is not automatically authoritative or current. The retrieval system must preserve these distinctions. ### Phase 15 — 📋 Prioritized Engineering Roadmap | Task | Priority | Work | Acceptance / rollback | | ---- | -------- | ----------------------------------------------- | --------------------------------------------------------------------------------------- | | T-01 | P0 | Instrument the full retrieval path | Every test query is traceable; rollback to existing logging if instrumentation fails | | T-02 | P0 | Build a labeled benchmark | Required evidence units and query segments are reviewed | | T-03 | P1 | Audit exact-ID ingestion and lookup | Exact-ID benchmark meets its predefined target; revert unverified normalization changes | | T-04 | P1 | Compare dense, BM25 and fusion outputs | Selected fusion meets quality gates; restore baseline fusion if regressions occur | | T-05 | P1 | Sweep candidate depths | Select a measured quality-cost configuration; restore previous K values if gates fail | | T-06 | P2 | Test structure-aware chunking | Improved evidence coverage with acceptable latency/index growth | | T-07 | P2 | Evaluate reranker and wider pool | Improved ranking without segment-level regressions | | T-08 | P2 | Add index freshness and metadata reconciliation | Required coverage and permission checks pass | | T-09 | P3 | Add adaptive routing and caching | Demonstrated benefit without stale evidence or access leakage | | T-10 | P0 | Add regression and release gates | All mandatory quality, security and latency tests pass | Owner roles: Retrieval engineer for search and ranking; data/ingestion engineer for parsing and indexing; evaluation engineer for benchmark quality; security engineer for authorization; platform engineer for latency and reliability. P0 means foundational work required before making trustworthy claims about improvement. P1–P3 indicate proposed sequencing, not measured business impact. ### Phase 16 — 🏁 Final Retrieval Engineering Blueprint ### 16.1 Executive summary The sample system uses dense retrieval and BM25 over PDF manuals and policies. Three reported symptoms deserve investigation: exact policy ID misses, low-ranked relevant results and incomplete multi-section evidence. The strongest recommendation is instrument first, then establish a labeled baseline. The supplied evidence does not yet identify whether the primary problem is corpus coverage, extraction, indexing, query processing, fusion, reranking or final context selection. ### 16.2 Recommended initial settings to test | Setting | Starting experiment | | -------------------- | ---------------------------------------------------------------------------- | | Dense candidates | 20 | | BM25 candidates | 20 | | Fusion | RRF baseline | | Fused pool | 20 | | Reranker pool | 20 | | Final context | Up to 5 chunks, with completeness checks | | Chunking | Keep 500/50 as baseline; test structure-aware alternative | | Query transformation | Original query first; targeted exact-ID lookup and decomposition experiments | | Evaluation | Labeled, segmented benchmark with fixed corpus version | These are experimental settings, not deployment-approved values. ### 16.3 Release validation gates * Relevant-evidence recall meets the predefined threshold. * Exact policy ID retrieval meets its predefined threshold. * Multi-section evidence coverage meets its predefined threshold. * Precision and ranking quality do not regress beyond allowed tolerances. * No-answer queries do not create unjustified confidence. * Tenant isolation and document-level authorization tests pass. * Source IDs, versions and provenance are preserved. * p95/p99 latency and cost satisfy documented service requirements. * All known regressions have an owner and documented disposition. * Rollback to the prior configuration is tested. ### 16.4 Final Action Blueprint 🚀 1\. DIAGNOSE Trace failed policy-ID queries through source documents, extracted text, index records, dense search, BM25, fusion, reranking and final context selection. 2\. ESTABLISH BASELINE Build a reviewed benchmark with exact-ID, multi-section, paraphrase and no-answer queries. Measure recall, ranking, evidence coverage, latency and cost. 3\. RUN THE FIRST EXPERIMENT Compare the existing retrieval configuration with a wider candidate pool and RRF fusion, keeping chunking, query processing and reranker behavior fixed. Use the existing configuration as the control. ### Final test report | Deliverable | Result | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | Three highest-priority actions | Instrumentation, benchmark creation, controlled retrieval experiment | | Required diagnostic evidence | Stage-level candidate IDs, ranks, scores, filters, source versions and latency | | Proposed first experiment | Current configuration vs. wider candidate retrieval with RRF | | Acceptance criteria | Predefined recall, exact-ID and evidence-coverage targets; acceptable precision, latency, cost and zero authorization violations | | Unresolved blockers | Missing logs, relevance labels, vendor capabilities and service-level targets | | Test execution status | Design completed; no live system tested | ### Final Readiness Status: CONDITIONALLY READY The blueprint is ready to guide diagnostic implementation and controlled experiments. Production approval is pending actual telemetry, benchmark results, verified security controls and successful regression testing. Final success criterion: Retrieve the most relevant, complete, authorized and verifiable evidence at acceptable cost and latency—not simply the largest number of documents. 😊