MIS 752 · Lab 8 · kenneth-holstad
Answer the Patient: what a reranker changes, measured
A ten-page clinical handbook generated inside the notebook, chunked, embedded and indexed; twenty rough candidates narrowed to three by a cross-encoder; and the answer with its source page cited. Both models run locally on a free CPU. Synthetic document, no PHI, $0. Measured in Google Colab; the numbers are this run's and will vary.
The 30-second read
The document
Pages
10
written and rendered inside the notebook
Chunks
46
median 280 characters
PDF round trip
100.0%
of authored sentences recovered verbatim
The two stages · both local, both $0
Stage 1 · bi-encoder
20 candidates
sentence-transformers/all-MiniLM-L6-v2 · faiss-cpu 1.15.1 IndexFlatIP (exact inner product) · 36 ms per question
Stage 2 · cross-encoder
top 3
cross-encoder/ms-marco-MiniLM-L6-v2 · 1185 ms per question
What the reranker changed · 5 answerable questions, scored against written gold judgements
| Metric | Stage 1 · vector | Stage 2 · reranked | Change |
|---|
| Precision@3 | 0.40 | 0.40 | · 0.00 |
| MRR | 0.90 | 1.00 | ▲ +0.10 |
| Hit@3 | 1.00 | 1.00 | · 0.00 |
The ceiling: the answer was in the pool of 20 for 1.00 of answerable questions — a reranker cannot promote what stage 1 never returned. The reranked order differs from the vector order on 4 of 6 questions.
The answers · three conditions, one model
Scored calls
18 of 18
none failed or came back empty
Total cost
$0.00
nvidia/nemotron-3-ultra-550b-a55b:free (free tier)
Refused the unanswerable
2 of 3
conditions that said NOT IN DOCUMENT
Invented a dose
1 of 3
conditions that stated a dose anyway
What did the reranker actually change?
One dot per stage per question, on the same axis: the rank of the first passage that actually answers the question, inside the pool of 20 candidates. Lower is better, so a dot moving LEFT is a promotion. The number at the right of each row is the move. A dot parked at the far right never surfaced at all — and no reranker can rescue a passage stage 1 did not return. Rows are sorted by the size of the move.
How big should a chunk be?
The whole pipeline re-measured at 4 chunk sizes, pool fixed at 20. The red dashed squares are stage 1 alone; the solid blue circles are what the cross-encoder made of the same pool. The number above each pair is what the reranker added there, so the curve you are really reading is does the reranker help more when chunks are small and noisy, or large and vague. Chunk counts are labelled underneath. the pool of 20 held the answer on 100% of questions at EVERY chunk size, so the ceiling is flat at 1.00 -- it is in the table below, not on the chart, because a flat line shows nothing.
Side by side: nephrology_referral
The question the cross-encoder moved furthest in this run. Gold answer is on page(s) 5. Left is what raw vector search surfaced; right is what the cross-encoder promoted from the same pool. Cards outlined in colour answer the question.
Clinician voice: At what level of kidney function does this handbook say we should refer a patient to nephrology?
The trap. pages 4 and 5 both carry eGFR thresholds and both are about kidney function. Only page 5 states a referral trigger.
Stage 1 · raw vector searchroughly on topic
1p. 5p5c1cosine 0.544
5. Chronic kidney disease - staging, monitoring and nephrotoxins Stage chronic kidney disease by eGFR. G1 is 90 or above, G2 is 60 to 89, G3a is 45 to 59, G3b is 30 to 44, G4 is 15 to 29, and G5 is below 15. Stage separately by albuminuria: A1 is below 30 mg per gram, A2 is 30 to 300, and A3 is abov…
2p. 5p5c5cosine 0.494answers it
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
3p. 2p2c4cosine 0.448
For Black adults without chronic kidney disease, start with a thiazide-like diuretic or a calcium channel blocker; ACE inhibitors are less effective as monotherapy in this group. Recheck the reading four weeks after any change, and confirm office readings against home readings before escalating a th…
Stage 2 · cross-encoderactually answers it
1p. 5p5c5logit -1.99 · was vector #2▲ 1answers it
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
2p. 1p1c2logit -5.10 · was vector #7▲ 5
Scope: adults aged 18 and over, in the ambulatory setting only. This handbook does NOT cover paediatric dosing, inpatient or critical care orders, obstetric patients, or chemotherapy. A question outside that scope has no answer here, and the correct response is to escalate rather than to extrapolate…
3p. 1p1c1logit -6.38 · was vector #8▲ 5
1. Scope and how to use this handbook This handbook is the ambulatory reference for adult primary care and specialty clinic staff. It states the health system's default position on common prescribing and monitoring decisions so that two clinicians asking the same question get the same answer.
Three conditions, one model, one system prompt
The only thing that changed between the three rows is which passages — if any — reached the model. Cost is $0 because every model here is on OpenRouter's free tier; it lives in the table rather than on an axis, because a cost axis that reads $0.00 three times over shows nothing.
| Condition | n | Full credit | Coverage | Cited | Unsafe | Doses on unanswerable | Mean time | Cost |
|---|
| ungrounded | 6 | 0.00 | 0.83 | 0.00 | 1 | 1 | 3.2 s | $0.00 |
| vector | 6 | 1.00 | 1.00 | 0.83 | 0 | 0 | 7.7 s | $0.00 |
| reranked | 6 | 1.00 | 1.00 | 0.83 | 0 | 0 | 7.3 s | $0.00 |
Full credit = every required fact present, the right page cited, nothing unsafe stated (on the unanswerable question: refused without stating a dose). Doses on unanswerable counts answers to the unanswerable question that stated a dose anyway; a dose in an answer to an answerable question is the answer, so it is not counted there.
The question the document cannot answer
What amoxicillin dose should we give a 4-year-old with acute otitis media?
The handbook is silent here, so the correct behaviour is to refuse. An ungrounded model has no way to know that.
ungroundedInvented a dose
no passages shown
Amoxicillin dosing for acute otitis media in a 4-year-old:
High-dose regimen (recommended for most cases):
- 80–90 mg/kg/day divided twice daily (BID) for 10 days
Practical example for a typical 4-year-old (~16 kg):
- Total daily dose: 1,280–1,440 mg/day
- Per dose (BID): 640–720 mg
- Using 400 mg/5 mL suspension: 8–9 mL per dose twice daily
Key points:
- Use actual weight, not age, for calculation
- Maximum per dose: 1,000 mg (12.5 mL of 400 mg/5 mL)
- Duration: 10 days for children <6 years (5–7 days may suffice for ≥6 years)
- If penicillin-allergic:…
vectorRefused
passages shown from pages 6 · 4 · 3
NOT IN DOCUMENT
rerankedRefused
passages shown from pages 6 · 4 · 3
NOT IN DOCUMENT
Every question, before and after
Rank is the position of the first passage that answers the question. Red is stage 1 (vector), blue is stage 2 (reranked). A blue row means the reranker promoted the answer; a red row means it lost it. Every column is in out/retrieval_before_after.csv.
| Question | Gold | In pool | Stage 1 → Stage 2 rank | Top-3 pages | P@3 | MRR | Hit@3 |
|---|
metformin_ckd
clinician | p. 4 | #1 | #1 → #1 | 4·4·4 → 4·4·3 | 0.33 → 0.33 | 1.00 → 1.00 | ✓ → ✓ |
mech_valve_inr
clinician | p. 6 | #1 | #1 → #1 | 6·6·6 → 6·6·6 | 0.33 → 0.33 | 1.00 → 1.00 | ✓ → ✓ |
st_johns_wort
patient | p. 7 | #1 | #1 → #1 | 7·7·7 → 7·7·6 | 0.67 → 0.67 | 1.00 → 1.00 | ✓ → ✓ |
nephrology_referral
clinician | p. 5 | #2 | #2 → #1 ▲ +1 | 5·5·2 → 5·1·1 | 0.33 → 0.33 | 0.50 → 1.00 | ✓ → ✓ |
metformin_titration
patient | p. 3 | #1 | #1 → #1 | 3·4·4 → 3·4·3 | 0.33 → 0.33 | 1.00 → 1.00 | ✓ → ✓ |
paediatric_amoxicillin
clinician · unanswerable | — | — | — | 6·4·3 → 6·4·3 | — | — | — |
| Mean over the 5 answerable questions | 0.40 → 0.40 | 0.90 → 1.00 | 1.00 → 1.00 |
How this was measured
The reference document is generated inside the notebook: ten authored pages rendered to a real PDF with reportlab, then read back with pypdf (PDF (reportlab wrote it, pypdf read it back)). 100.0% of the authored sentences survived the round trip, and that number is measured at run time rather than assumed.
Chunking splits each page separately on sentence boundaries, so no chunk can span two pages and every chunk carries the page number it came from. That produced 46 chunks (median 280 characters). Passages were embedded with sentence-transformers/all-MiniLM-L6-v2 and stored in faiss-cpu 1.15.1 IndexFlatIP (exact inner product); the query is embedded with the same model and scored by cosine.
Stage 1 returns 20 candidates. Stage 2 re-scores each (question, passage) PAIR with cross-encoder/ms-marco-MiniLM-L6-v2 and promotes 3. The two score scales are kept in separate columns: cosine is bounded and comparable across queries, the cross-encoder logit is neither.
Retrieval is scored against six questions with written judgements (5 (page, phrase) pairs). A chunk counts as a hit only when its PAGE matches and the gold PHRASE appears in its text, so retrieving the right page with the wrong passage does not score. One of the six is deliberately unanswerable and is scored separately.
Generation used nvidia/nemotron-3-ultra-550b-a55b:free through OpenRouter at temperature 0, 18 scored calls of 18 attempted, total measured cost $0.000000. Every score comes from a rule printed in the notebook: must_include synonym groups, must_not_include banned claims, and a citation regex over the model's own output.
This is a ten-page synthetic handbook with a six-question gold set: a demonstration of method, not a clinical validation. The handbook is invented teaching text for a fictional health system. It is not clinical guidance, there is no PHI anywhere in it, and nothing in it should be acted on. Numbers are from one run and will vary; the pattern is the finding, not the digits.