# RAG Flamegraph Profiler™ v1.1 --- ## ▶ PROFILE VERDICT **🟠 DEGRADED** `[████░░░░░░] 4/10` The correct chunk was retrieved prominently — c2 (0.88 cosine, contains the 90-day figure) ranks second-highest of all ten chunks — but two bulk, tangential chunks (c3: general termination provisions, 1,740 tok; c6: definitions/boilerplate, 1,920 tok) together account for 41% of total context tokens despite mid-to-low relevance. With no re-ranker to demote them, the specific figure in c2 was present in context but generation defaulted to vague phrasing. This is a **NEEDLE-IN-HAYSTACK VOID**, compounded by a **LONG TAIL** that likely didn't help. Highest-leverage fix: reduce Top-K toward the 3 chunks the evidence actually supports, or insert a re-ranker/compressor — before touching embedding model or chunking strategy. --- ## ▶ METRIC SELECTION (W.I.D.T.H. Protocol™) **Rule fired: W1** — friction=relevance, scores present. **Sub-rule: N1** — single score family (all cosine). Min–max normalized within this run to 0–100. Raw min = 0.31 (c10), raw max = 0.92 (c1) — invert against these if comparing chunks by eye. Data census: all cosine scores fall within valid [0,1] range — no N5 exclusions. No timing data present (expected; friction is relevance, not latency). | chunk | raw cosine | tokens | width (0–100) | |---|---|---|---| | c1 | 0.92 | 460 | 100 | | c2 | 0.88 | 520 | 93 | | c3 | 0.71 | 1,740 | 66 | | c4 | 0.49 | 680 | 30 | | c5 | 0.46 | 710 | 25 | | c6 | 0.43 | 1,920 | 20 | | c7 | 0.39 | 640 | 13 | | c8 | 0.36 | 590 | 8 | | c9 | 0.34 | 810 | 5 | | c10 | 0.31 | 760 | 0 | --- ## ▶ FLAMEGRAPH RENDER ``` chunk layer [c1 ████████████████████ 100] cancellation clause (cos=0.92, 460tok) [c2 ███████████████████░ 93] notice period: 90 days (cos=0.88, 520tok) [c3 █████████████░░░░░░░ 66] general termination (cos=0.71, 1,740tok) [c4 ██████░░░░░░░░░░░░░░ 30] subscription renewal (cos=0.49, 680tok) [c5 █████░░░░░░░░░░░░░░░ 25] service suspension (cos=0.46, 710tok) [c6 ████░░░░░░░░░░░░░░░░ 20] definitions/boilerplate (cos=0.43, 1,920tok) [c7 ███░░░░░░░░░░░░░░░░░ 13] payment disputes (cos=0.39, 640tok) [c8 ██░░░░░░░░░░░░░░░░░░ 8] data-processing appendix (cos=0.36, 590tok) [c9 █░░░░░░░░░░░░░░░░░░░ 5] support obligations (cos=0.34, 810tok) [c10░░░░░░░░░░░░░░░░░░░░ 0] limitation of liability (cos=0.31, 760tok) ────────────────────────────────────────────────────────────────────── search layer [ dense (Qdrant): all 10 chunks — 8,830 tok total, no re-ranker ] decomp layer [ sq1: "enterprise agreement cancellation notice period" — no rewriting ] pipeline root [ user query: cancellation notice period ] ``` **d3-flame-graph JSON:** ```json { "name": "root: user query", "value": 8830, "children": [ { "name": "decomposition: sq1 (no rewriting)", "meta": {"layer": "decomposition", "raw_score": null, "tokens": null, "derived": false}, "children": [ { "name": "search: dense (Qdrant, no re-ranker)", "value": 8830, "children": [ {"name": "chunk_1", "value": 460, "meta": {"layer":"chunk","raw_score":0.92,"tokens":460,"derived":false,"width_n1":100}}, {"name": "chunk_2", "value": 520, "meta": {"layer":"chunk","raw_score":0.88,"tokens":520,"derived":false,"width_n1":93}}, {"name": "chunk_3", "value": 1740, "meta": {"layer":"chunk","raw_score":0.71,"tokens":1740,"derived":false,"width_n1":66}}, {"name": "chunk_4", "value": 680, "meta": {"layer":"chunk","raw_score":0.49,"tokens":680,"derived":false,"width_n1":30}}, {"name": "chunk_5", "value": 710, "meta": {"layer":"chunk","raw_score":0.46,"tokens":710,"derived":false,"width_n1":25}}, {"name": "chunk_6", "value": 1920, "meta": {"layer":"chunk","raw_score":0.43,"tokens":1920,"derived":false,"width_n1":20}}, {"name": "chunk_7", "value": 640, "meta": {"layer":"chunk","raw_score":0.39,"tokens":640,"derived":false,"width_n1":13}}, {"name": "chunk_8", "value": 590, "meta": {"layer":"chunk","raw_score":0.36,"tokens":590,"derived":false,"width_n1":8}}, {"name": "chunk_9", "value": 810, "meta": {"layer":"chunk","raw_score":0.34,"tokens":810,"derived":false,"width_n1":5}}, {"name": "chunk_10", "value": 760, "meta": {"layer":"chunk","raw_score":0.31,"tokens":760,"derived":false,"width_n1":0}} ] } ] } ] } ``` **Collapsed-stack format:** ``` root;sq1;dense;chunk_1 100 root;sq1;dense;chunk_2 93 root;sq1;dense;chunk_3 66 root;sq1;dense;chunk_4 30 root;sq1;dense;chunk_5 25 root;sq1;dense;chunk_6 20 root;sq1;dense;chunk_7 13 root;sq1;dense;chunk_8 8 root;sq1;dense;chunk_9 5 root;sq1;dense;chunk_10 0 ``` --- ## ▶ SHAPE DIAGNOSIS **1. NEEDLE-IN-HAYSTACK VOID — confidence HIGH** Depends on: chunk layer + generation outcome. - *Render pass*: c2 (width 93) sits second-widest of ten blocks — visually prominent, not buried at the tail. - *Raw-number pass*: c2's raw cosine (0.88) is second-highest of all ten raw scores, 0.04 below c1. - Both passes agree → verdict stands. The 90-day figure was retrieved into a highly-ranked chunk, but c3 (1,740 tok) and c6 (1,920 tok) — together 3,660 of 8,830 total tokens (41%) — are large-bodied and lower-specificity by content description ("general termination provisions", "definitions and legal boilerplate"). With no re-ranker or compression stage, these bulk blocks are the most plausible dilution source causing generation to default to vague phrasing over the specific figure sitting in c2. **2. FRACTURED FLOOR — N/A** Depends on: decomposition layer. Only one sub-query was issued, no rewriting occurred — the opposite of over-splintering. This shape does not apply; not a contributing failure mode here. **3. RE-RANKER DROP-OFF — confidence HIGH (structural absence)** Depends on: search layer + chunk layer. [PIPELINE_CONTEXT] confirms no re-ranker. As in prior runs, this isn't the "re-ranker working" shape (none exists to show it) — it's the structural condition that lets #1 happen unchecked: nothing compresses the 10-chunk mixed-relevance layer before generation sees it. **LONG TAIL check — confidence MED (Analysis)** Width drops sharply and monotonically after c3: 100 → 93 → 66 → **30** → 25 → 20 → 13 → 8 → 5 → 0. The gap from c3 (66) to c4 (30) is more than double, and c4 onward shifts topically away from cancellation entirely (renewal, suspension, boilerplate, payment disputes, DPA, liability — none directly on-topic). This matches the long-tail pattern: a few wide relevant blocks (c1–c3) followed by many narrow, off-topic blocks. **Evidence supports Top-K ≈ 3** for this query shape (Analysis — reasoned from this run, not measured across runs). Not a hard fix recommendation; test against the CALIBRATION CHECK next run. --- ## ▶ VERDICT GATE — DECISION LOOP - G1 (findings cite named blocks): PASS — c1–c10 referenced by label throughout. - G2 (recommendations trace to findings): PASS — Top-K≈3 traces to LONG TAIL; re-ranker/compression traces to findings #1/#3. - G3 (confidence per finding): PASS — HIGH/N-A/HIGH/MED assigned with limitation stated for each; Top-K figure marked (Analysis). - G4 (scores+timings both absent → cap DEGRADED-suspected): N/A — scores present, no cap triggered. - G5 (≤3,500 tokens): PASS. `LOOP: iteration 1/2 → PASS` ``` --- LOOP STATE (paste into [PREVIOUS_RUN_NOTES] next time) --- run_date: 2026-07-20 | verdict: DEGRADED 4/10 findings: needle-in-haystack void=HIGH (c3+c6 bulk vs. narrow c2); re-ranker absence=HIGH (structural); fractured floor=N/A; long tail=MED (Analysis, Top-K≈3 suggested, not applied) predictions: cutting Top-K to ~3–4 OR adding a re-ranker/compressor before generation → c3/c6 width should shrink or exit the post-processing layer, and the 90-day figure from c2 should appear explicitly in the generated answer next_run: apply chosen fix, re-profile the same query, paste this block + what you changed -------------------------------------------------------------- ``` --- ## ▶ READING GUIDE - **Look first at**: whether c2's content (or its post-fix equivalent) survives verbatim into the generated answer — that's the ground truth this whole profile is diagnosing toward, more than any width number. - **Shape-change that means it worked**: the chunk layer narrows to ~3–4 blocks, c3/c6-style bulk blocks are absent or heavily trimmed, and the specific 90-day figure appears in generation. - **Shape-change that means a new problem emerged**: if c1/c2 width also drops sharply after the fix (over-aggressive Top-K cut or over-compression removing relevant content, not just bulk) — that's a new failure mode, not a resolved one.