AI-generated illustrationRetrieval That Can Be Checked: RAG, Search, and Citations
Build retrieval whose answers a program can check: the right revision for this person and date, claims tied to exact passages, and an honest abstention.
- Reading
- 58 min
- Workshop
- 115 min
- Stack
- TS · Node.js 22 · pnpm 12 · BM25, bge-small-en-v1.5 and rank fusion · optional Claude API (@anthropic-ai/sdk 1.12.0)
- Version
- 1.0.0
- Last verified
- Nov 21, 2026
01Outcomes
What you will build and be able to do
Each outcome has an observable criterion. The lab and the assessment check exactly these, nothing more and nothing less.
OBJ-001
Find the passage a claim needs
Find the passage a claim needs: apply one shared eligibility filter (approved status, a half-open effective interval, and scope) and an identifier-preserving tokenizer, so the right revision and the exact identifier rank in the top k. Checked by pnpm lab1:check and ASM-001.
OBJ-002
Compare retrievers on their own scorecard
Compare BM25, semantic, and hybrid retrieval at a fixed k against a direct-document baseline, and read the retrieval scorecard separately from answer verdicts. Checked by pnpm lab2:check and ASM-001.
OBJ-003
Return an answer a program can check
Return a checkable answer: atomic claims whose references resolve to a document revision, passage, and character range, verified for existence, range, quote fidelity, delivered provenance, effective date, scope, support, responsiveness, and authority. Checked by pnpm lab3:check and ASM-002.
OBJ-004
Abstain with a reason and a next step
Abstain with a specific reason and a next step when no eligible support exists, the scope is ambiguous, or a conflict has no policy resolution, without abstaining on questions the library can answer. Checked by the LAB-003 positive control and ASM-002.
OBJ-005
Decide from a rule written first
Defend the simplest approach that meets predeclared quality and safety thresholds, or hold and name the evidence that would change the decision. Checked by pnpm lab2:check and ASM-003.
02Prerequisites and setup
Unpack the download and check your environment ($0.00)
- Evaluation basics. You can read a code grader and know why a run should be repeatable. If not, start with Agent Evaluations That Measure Real Outcomes.Start with Guide 02
- Context versus record. You know that what a model has in its context is not the system of record. Context Engineering for Long-Running Agents covers it.Read Guide 08
- TypeScript and JSON. The kit is TypeScript run directly by Node.js; every rule translates to Python, Go, or Rust.
- Node.js 22.18 or newer and pnpm 12. The download pins Node.js 22.23.5 (.node-version, Node.js release index), the release line where TypeScript type stripping is on by default from 22.18.0 (Node.js 22 docs), and pnpm 12.21.0 (npm registry).pnpm installation guide
- No API key. Every lab, test, and the assessment run offline at $0.00. One optional command sends one question to Claude with your own key.
unzip -q retrieval-labs.zip -d retrieval-labs-workcd retrieval-labs-work/labs/startercorepack enablepnpm install --frozen-lockfilepnpm preflight03The system picture
Three jobs, three scorecards

Extended description
Title: "Three jobs, three scorecards." A diagram in two rows of three columns.
Top row, three solid cards joined left to right by arrows: "Retriever", "proposes evidence": "question, as_of, and scope"; "eligibility filter"; "rank, keep top 5"; "Answerer", "makes claims": "claim: value and its conditions"; "reference: revision, passage, and range"; "Checker", "licenses claims": "9 checks per claim", then two outcome boxes: mint "answer: licensed claims" and amber "abstain: reason and next step".
Bottom row, three dashed blue cards, one under each job, naming the scorecard that job is graded on: "Retrieval": "recall@5, MRR, context size"; "Answers": "correct supported, citations valid vs supporting"; "Abstention": "correct abstentions, false abstentions".
The point of the figure: each job has its own scorecard, so a retrieval score never stands in for an answer score, and a valid citation is counted apart from a supporting one.
04Lesson
Three jobs, three verdicts
A retrieval-augmented answer can fail in three different places, and each needs its own evidence:
- The retriever proposes evidence. It filters the library to what is eligible for this question's date and scope, ranks it, and delivers a short list of passages. Its job is to include what the answer needs; it does not decide what is true.
- The answerer makes claims. Each claim is one decisive statement with a value, its conditions, and a reference to the revision, passage, and character range it came from.
- The checker decides which claims the evidence licenses. It verifies each reference against the stored corpus, not against the answerer's prose. A claim that fails is dropped; an answer with nothing left becomes an abstention with a reason and a next step.
Keeping the jobs apart is what makes failures diagnosable. In this guide's measurements retrieval reached every required claim, yet the decision was still hold; a single blended accuracy number would have hidden where the problem was.
Eligibility before ranking
A policy library is not a pile of text. Each document revision has a status (approved or draft), an effective interval, a scope (region and worker type), and an authority level: policy and standard 1, procedure 2, guidance 3. The home office allowance shows why all of them matter. EQP-100 revision 1 set a one-time USD 500 stipend until 2026-07-01. Revision 2 replaced it from that day with up to USD 600 every 36 months. Revision 3 raises it to USD 750 from 2027-01-01. An approved FAQ, guidance with no end date, still says USD 500.
Three rules decide eligibility, and they apply before any ranking:
- Status: drafts are never eligible.
REC-SCHEDr8, a draft, would changeRC-14/Bretention from 7 years to 8; until it is approved, r7 governs. - Effective interval, half-open: a revision governs from its first day up to, but not including, its end date:
effective_from <= as_of < effective_to. On the day one revision ends and the next begins, exactly one governs. - Scope: a request for an EU employee cannot be answered from the US policy, however similar the text.
Extended description
Title: "One policy, three revisions." Subtitle: "Each interval includes its first day and excludes its end date." A timeline with year ticks at 2025, 2026, and 2027 and four bars, one per document revision, each labeled at the left in monospace: "EQP-100 r1": a bar outlined in light gray from 2025-01-01 to 2026-07-01, labeled "USD 500, one time: superseded"; "r2": a mint bar from 2026-07-01 to 2027-01-01, labeled at its left "USD 600, every 36 months: in effect"; "r3": a dashed blue outline from 2027-01-01 to the right edge, with no end date, labeled "USD 750, every 36 months: not yet effective"; "FAQ-EQP r1": a dashed amber outline from 2025-03-01 to the right edge, with no end date, labeled "guidance, USD 500, one time: outranked by policy".
Three vertical date lines, labeled in monospace at the top: "D03" at 2026-06-30, "D02" at 2026-07-01 (one day later, so the two lines nearly coincide), and "D01" at 2026-11-01. A legend below the axis reads "D03 2026-06-30: r1", "D02 2026-07-01: r2", and "D01 2026-11-01: r2", the revision that governs each question's as_of date. Footer: "Synthetic data. Simulated services."
What it shows: the newest text is not always the governing text. On 2026-06-30, r1 still governs; on its first day, 2026-07-01, r2 governs; r3 does not govern until 2027-01-01; and the FAQ, a guidance page at a lower authority level, never outranks the policy.
Animated version (silent, 410 frames at 30 fps, 13.7 s): the four bars grow in from the left, one every 22 frames from frame 18. From frame 130 the three date lines appear one every 70 frames, D03, then D02, then D01; while each date line is the newest, the bars that do not govern on that date dim, so only the governing revision stays bright. From frame 340 every bar is at full strength, and the last 70 frames hold on the still above, which is the poster. Nothing flashes. On the site the clip loops silently while it is in view and has a Play and Pause control; with reduced motion requested, it does not start on its own and the poster stays until the reader presses Play animation.
Apply the filter once, in one place, to every retrieval mode. The starter kit's retriever prints filter eligible while ranking all 88 passages, and for the stipend question BM25 puts a superseded revision second. A filter applied in one index only is the most common partial repair, and it leaves the other modes ranking stale text.
Filtering is necessary and not sufficient. After the repair, each condition still delivers four disqualifying passages across the dev questions. The FAQ is the clearest case: it is approved, in effect, and in scope, so no filter removes it, and only an authority check against every eligible source for the same rule can say the policy outranks it. That check belongs to the checker (The checker: nine checks).
Set up the kit
Unpack the download (retrieval-labs.zip), install once from the lockfile (this needs network), and run preflight. Everything after the install runs offline.
unzip -q retrieval-labs.zip -d retrieval-labs-workcd retrieval-labs-work/labs/startercorepack enablepnpm install --frozen-lockfilepnpm preflightRecorded output on a clean extract (2026-11-21):
ok Node.js 22.23.5 (needs 22.18 or newer to run TypeScript directly)ok corpus: 21 documents, 88 passages, 82 register facts (synthetic: Example Instruments)ok frozen vectors: 102 loaded, 0 passages without a vector (BAAI/bge-small-en-v1.5@5c38ec7)ok shipped inputs and grader files match data/MANIFEST.jsonok run outputs go to runs/ (never data/)preflight: ready (offline, no API key needed)LAB-001: retrieve the right revision (about 35 minutes)
In the hands-on labs, start from the starter's ranking for D01 (pnpm rank D01 --mode bm25) and repair three seeded faults: an eligibility filter that is computed and then ignored, an exclusive first day, and a tokenizer that drops digits. pnpm lab1:check starts at 2 of 6 and ends at 6 of 6; you also confirm that D02, asked on 2026-07-01, gets revision 2 without deleting revision 1.
Four ways to find evidence
The kit holds four retrieval conditions steady on the same corpus, passages, eligibility filter, questions, k = 5, and candidate depth 20 per index.
BM25 scores passages by term frequency, inverse document frequency, and length normalization. The kit follows Robertson and Zaragoza's formulation, in which the parameter b sets how much document length is normalized, from fully at 1 to not at all at 0 (Robertson and Zaragoza, 2009). It uses Lucene's conventional defaults, "k1 = 1.2 b = 0.75" (Lucene BM25Similarity). BM25 is strong on exact identifiers and weak when the question uses different words than the passage.
Semantic retrieval compares embeddings. "Anthropic does not offer its own embedding model" (Claude embeddings docs), so the kit uses the MIT-licensed BAAI/bge-small-en-v1.5, whose card gives the retrieval query prefix "Represent this sentence for searching relevant passages:" (model card). Vectors ship frozen, keyed by the hash of each exact input, so the labs need no model download; the optional tools/encoder regenerates them.
Hybrid retrieval fuses both ranked lists with reciprocal rank fusion, which scores each passage by the sum of 1 / (k + rank) over the lists; its authors' experiments "indicated that k = 60 was near-optimal" (Cormack, Clarke, and Buettcher, 2009), and the kit uses 60. Anthropic's contextual retrieval article describes the same shape at scale: "Combine and deduplicate results from (3) and (4) using rank fusion techniques" (Introducing Contextual Retrieval).
Direct reads every eligible passage, in document order, with no index. It is the baseline every ranked condition has to beat. The same article notes that a knowledge base under 200,000 tokens, about 500 pages, can go into the prompt whole (Introducing Contextual Retrieval). This library is far smaller, so direct is a real competitor, not a strawman.
Two dev questions show the trade (instructional examples from the dev split, after LAB-001):
D07asks how longRC-14/Brecords must be kept. With a tokenizer that drops digits,RC-14/Bbecomes the common fragmentrc, and the right passage ranks sixth. Keeping identifiers whole moves it to rank 1, with the confusable neighborRC-14/Asecond.D01asks about the "home office stipend". The current policy no longer uses the word "stipend", so BM25 ranks it sixth, outside k = 5, while the old FAQ that still uses the word ranks first. Semantic retrieval finds the policy at rank 2.
Contextual retrieval, as Anthropic describes it, "solves this problem by prepending chunk-specific explanatory context to each chunk before embedding" (Introducing Contextual Retrieval). The kit does a small version of that: each passage's index text carries its document title, scope, and section heading. This guide does not run the full technique or import the article's reported gains; test it on your own questions.
Measure retrieval on its own scorecard
Score retrieval at claim grain, not document grain. A question can need two or three claims from different passages, and "the right document appeared" says nothing about whether the second passage did. The kit reports, for answerable questions only:
| Metric | What it answers |
|---|---|
| Claim recall at k | Of the required claims, how many had acceptable evidence in the top k |
| Full coverage at k | How many questions had every required claim covered |
| Multi-passage coverage | The same, for questions that need more than one passage |
| MRR at depth | How high the first acceptable passage ranks (reciprocal rank, averaged) |
| Delivered disqualifying | How many delivered passages are known wrong evidence: stale, out of scope, or outranked |
| Context characters | How much text the answerer has to read |
On the dev split (instructional, from the labs), LAB-001's repair changes every column:
| Dev split, k = 5 | Claim recall | Disqualifying delivered | Eligible pool | Mean context characters |
|---|---|---|---|---|
| BM25, starter | 17/23 | 16 | 88.0 | 1,486 |
BM25, after LAB-001 | 22/23 | 4 | 49.7 | 1,397 |
Semantic, after LAB-001 | 22/23 | 4 | 49.7 | 1,468 |
Hybrid, after LAB-001 | 23/23 | 4 | 49.7 | 1,475 |
Direct, after LAB-001 | 23/23 | 4 | 49.7 | 14,518 |
On the frozen held-out split eval2 (24 questions, run once on 2026-11-21), the four conditions tie on retrieval:

eval2.Extended description
Title: "Retrieval tied at 19 of 19." Two panels from the frozen held-out run FREEZE-2 on split eval2, 2026-11-21, with the extractive answerer.
Left panel, "Claim recall at 5", four full mint bars: "Direct*": "19 of 19"; "BM25": "19 of 19"; "Semantic": "19 of 19"; "Hybrid": "19 of 19".
Right panel, "Mean characters read per question", four blue bars drawn to scale: "Direct": "14,579" (a full-width bar); "BM25": "1,318"; "Semantic": "1,414"; "Hybrid": "1,404".
Note under the panels: "* Direct reads every eligible passage, unranked." Amber box: "After the checker, every condition had 12 of 17 correct supported answers, under the 0.75 bar. Decision: hold." Footer: "Synthetic data. Simulated services. 2026-11-21."
What it shows: on this corpus every condition retrieved the passages for all 19 required claims, and the three ranked conditions did it while reading about a tenth of the text that direct reading did; the answer stage, not retrieval, is what kept the run from passing. Semantic retrieval used the kit's frozen BAAI/bge-small-en-v1.5 vectors. The tie holds for one small synthetic corpus and these 19 claims; it does not rank the strategies in general.
| Condition | Claim recall at 5 | Full coverage | Multi-passage | MRR | Mean context characters |
|---|---|---|---|---|---|
| Direct | 19/19 | 17/17 | 2/2 | not ranked | 14,579 |
| BM25 | 19/19 | 17/17 | 2/2 | 0.971 | 1,318 |
| Semantic | 19/19 | 17/17 | 2/2 | 0.971 | 1,414 |
| Hybrid | 19/19 | 17/17 | 2/2 | 0.971 | 1,404 |
Hybrid retrieval scored a union of 22.6 candidates per question to deliver its five. The same frozen run also scored every condition with the eligibility filter off, reported separately as deliberate stale retrieval: recall stayed 19 of 19, but six disqualifying passages were delivered and correct supported answers fell from 12 to 10 of 17. Recall alone would not have shown it.
When every retriever reaches every required claim, the retriever is not what decides the outcome. That is a finding about this corpus, not a law: 24 synthetic questions over 21 short documents, with denominators small enough that one question moves a rate by several points.
LAB-002: compare retrieval and decide (about 30 minutes)
In the hands-on labs, run pnpm compare, inspect a paraphrase (D01), an exact identifier with a confusable neighbor (D07), and a three-claim question (D13) with pnpm inspect, see what the filter buys with pnpm compare --stale, read the frozen scorecard with pnpm scorecard --frozen, and write work/recommendation.md. pnpm lab2:check confirms that the recommendation names a decision, quotes at least three counts from the frozen scorecard, states the safety rule, keeps retrieval apart from answer verdicts, and states its limits. It does not grade whether the decision is right; a peer or instructor does, against the rule.
A checkable answer contract
An answer is either atomic claims with specific references, or an abstention with a reason and a next step. Nothing else:
{ "decision": "answer", "claims": [ { "text": "RC-14/B vendor invoices must be kept for 7 years after the fiscal year in which the invoice was paid.", "value": "7 years", "qualifiers": ["after the fiscal year in which the invoice was paid"], "refs": [ { "doc_id": "REC-SCHED", "revision": "r7", "passage_id": "s2.p3", "start": 0, "end": 97, "quote": "RC-14/B, vendor invoices: retain for 7 years after the fiscal year in which the invoice was paid." } ] } ]}Every reference names a revision, not only a document, because the revision is what has an effective interval. Ranges are 0-based and end-exclusive on canonical passage text (NFC, LF line endings, collapsed whitespace), so any program can recompute them. A model supplies the quote; the application resolves it to a range in the stored text.
This is application-managed citation. Claude also has native citations, which return exact cited spans from documents you pass in, and "cited_text does not count toward your output tokens" (Claude citations docs). Two limits shaped this kit:
- Native citations and JSON outputs do not mix. The docs state that "citations require interleaving citation blocks with text output, which is incompatible with the strict JSON schema constraints of structured outputs" (Claude citations docs). On 2026-11-21 a request that enabled both returned HTTP 400 with "Citations cannot be enabled when output format is set". The optional live path uses JSON outputs through
output_config.format(structured outputs docs), so it cannot also use native citations. - A native citation is a pointer the API resolved, not proof. It shows where the text came from; it does not show that the text supports the claim for this person on this date.
Search result blocks let you "Get natural citations for RAG applications by providing search results with source attribution" (search results docs). They fit when you want Claude to cite retrieved passages in prose; the checker in this guide would still run on what comes back.
The checker: nine checks
The checker runs per claim, in order, and records every check it ran:
| Check | Blocks a claim when | Lab fixture |
|---|---|---|
| Exists | the reference names a passage that is not in the corpus | None |
| Range | the range is not integers with 0 <= start < end <= length | L3-D |
| Quote | the quote is not exactly the cited slice | L3-E |
| Provenance | the passage was not in the delivered context | None |
| Effective | the revision is a draft, superseded, or not yet effective on the request date | L3-C |
| Scope | the revision covers a different region or worker type | L3-G |
| Support | no register fact inside the cited range states the claim's value, unit, required qualifiers, and subject | L3-B, L3-F, L3-I |
| Responsive | the question names none of the supporting fact's subject terms (scope words such as "employees" do not count) | L3-K |
| Authority | a higher-authority eligible source governs the same rule, or same-authority sources disagree | None |
Then a gate turns verdicts into an outcome. A conflict on any claim abstains with conflict_unresolved; a claim whose value depends on a scope the question does not give abstains with scope_ambiguous; otherwise unlicensed claims are dropped, and an answer with nothing left abstains with no_eligible_support.
LAB-003, not model output. Silent loop, 11 seconds.Extended description
Title: "Where each seeded fault is caught." Subtitle: "7 of 11 lab fixtures; L3-A is the control." A vertical ladder of nine rungs, labeled in monospace from top to bottom: "exists", "range", "quote", "provenance", "effective", "scope", "support", "responsive", "authority". Each fixture's card sits on the rung of the one check its claim fails, with a cross, and a short note to its right: range: "L3-D", "[40, 400) is outside a 97-character passage"; quote: "L3-E", "quote differs from REC-SCHED r7 s2.p3 [0, 44)"; effective: "L3-C", "EQP-100 r1 was superseded on 2026-07-01"; scope: "L3-G", "TRV-US r4 covers US employees"; support: "L3-B", "the cited text says 90 days, not USD 600"; responsive: "L3-K", "true, but about remote work equipment allowance"; authority (the bottom rung): "L3-A" in mint with a check mark, "passes all 9 checks: licensed".
The rungs "exists" and "provenance" have no card. Footer: "Offline: scripted model double, not Claude."
What it shows: the checker runs all nine checks on every claim, and each faulty fixture fails exactly one of them, so a reader can see which check catches which kind of fault. L3-K's claim is true and correctly cited, but its question asks about pet relocation, so it is not responsive. These are constructed fixtures, not model output.
Animated version (silent, 322 frames at 30 fps, 10.7 s): the ladder is drawn first; from frame 24, one card every 34 frames slides onto its rung over 16 frames, in ladder order from L3-D at range to L3-A at authority, and its note fades in as the card lands. The last 60 frames hold on the still above, which is the poster. Nothing flashes. On the site the clip loops silently while it is in view and has a Play and Pause control; with reduced motion requested, it does not start on its own and the poster stays until the reader presses Play animation.
Worked example: a valid pointer that does not support the claim
Fixture L3-B answers D01 ("How much can an employee claim under the home office stipend?", as of 2026-11-01, for an employee) with one claim:
- Claim: "Employees can claim up to USD 600 every 36 months." (value
USD 600, qualifierevery 36 months) - Reference:
EQP-100revisionr2, passages3.p1, range[0, 69), quote "Submit the claim with an itemized receipt within 90 days of purchase."
Every pointer check passes. The passage exists, the range fits its 69 characters, the quote is the exact slice, the passage was delivered, revision 2 is in effect on 2026-11-01, and it covers employees in every region. Then support fails: the only register fact inside [0, 69) is the 90-day claim deadline, not an amount. The right document, the right revision, and an exact quote still do not license USD 600. With no licensed claim left, the gate abstains:
ok L3-B Valid pointer to the right document, wrong section (unrelated citation) claim 0: blocked - support: the cited text says 90 days, not "USD 600" (checks run: exists, range, quote, provenance, effective, scope, support, responsive, authority) gated: abstain:no_eligible_supportThe repair is a different reference, not a different sentence. Fixture L3-A cites passage s2.p1, range [0, 49), quote "Employees may claim up to USD 600 every 36 months". The register fact there states USD 600 with its required qualifier "every 36 months", the question names its subject ("home office", "stipend"), and the FAQ's USD 500 is guidance, outranked by the policy:
ok L3-A Supported answer citing the exact sentence span claim 0: licensed (checks run: exists, range, quote, provenance, effective, scope, support, responsive, authority) gated: answerThe same support check blocks a right value that loses its condition. L3-I claims "Contractor accounts expire after 90 days." from a passage that says they expire after 90 days "unless the sponsoring manager renews them", and is blocked with support: the claim drops the condition: unless the sponsoring manager renews. Scope works the same way: L3-G quotes the US travel policy's 30-day receipt rule exactly for an EU employee, and is blocked with scope: TRV-US@r4 covers US employees; the EU rule is 45 days.

L3-B and L3-A.Extended description
Title: "A valid pointer is not support." Two cards for the same claim, from lab fixtures L3-B and L3-A.
Top card, amber, "L3-B": "USD 600 every 36 months" "cites" "EQP-100 r2 s3.p1 [0, 69)". The cited text, quoted: "Submit the claim with an itemized receipt within 90 days of purchase." Check row: "exists", "range", "quote", "provenance", "effective", and "scope" each with a check mark; "support" with a cross; "responsive" and "authority" with check marks. Outcome: "abstain: no_eligible_support", and the reason "support: the cited text says 90 days, not USD 600".
Bottom card, mint, "L3-A": the same claim "USD 600 every 36 months" "cites" "EQP-100 r2 s2.p1 [0, 49)". The cited text, quoted: "Employees may claim up to USD 600 every 36 months". All nine checks with check marks. Outcome: "answer: licensed".
Footer: "Offline: scripted model double, not Claude."
What it shows: L3-B's reference is real, in range, quoted exactly, from the revision in effect, and in scope, so every pointer check passes; the claim still fails, because the cited words do not say what the claim says. Checking that a citation resolves is not checking that it supports.
Two properties matter more than any single check:
- Support is structured, not textual. The support check looks for a register fact (a stored value, unit, qualifiers, and subject) inside the cited range. Substring match would license an exact quote that states a different number, and a fluent model judge can agree with a confident sentence. Neither is entailment, and this kit uses neither as proof.
- The checker reads
valueandqualifiers, not the claim sentence. That is a deliberate scope limit with a measured cost, shown inF-14: a claim can be licensed on its structured fields while its sentence leaves the value out. Render display text from checked fields if the sentence is what users read.
LAB-003: build the checker (about 50 minutes)
In the hands-on labs, start from a checker that compares each quote with the whole passage and checks nothing else, and build the nine checks against the eleven constructed fixtures, L3-A to L3-K. pnpm lab3:check starts at 3 of 12 and ends at 12 of 12, and its last test is a positive control: a checker that abstains on everything must fail the suite.
Abstain with a reason
An abstention is an answer: a reason and a next step a person can act on. The kit returns these next steps:
| Reason | When | Next step |
|---|---|---|
no_eligible_support | No in-effect, in-scope passage supports an answer | "No in-effect passage supports an answer. Ask the policy owner, or rephrase with the policy or control name." |
scope_ambiguous | The answer depends on a region or worker type the question does not give | "The answer depends on scope. Say which region or worker type applies, for example US employees or EU employees." |
conflict_unresolved | In-effect policies disagree and the order of authority does not settle it | "In-effect policies disagree and the order of authority does not settle it. Ask the policy owners which rule governs." |
Abstention has two failure modes, and the scorecard counts both: answering a question the library cannot answer, and abstaining on one it can. A checker that refuses everything is perfectly safe and useless. In the assessment's calibration set, a blanket-abstention submission scores Emerging on checkable answers and fails. The decision rule in Decide with a rule written first caps false abstentions at 0.20 of answerable questions and requires correct supported answers of at least 0.75, so safety bought with refusals does not pass.
What the measurements showed
Everything in this section is measured, not illustrative. The decision rule was written before any held-out run. Each frozen run used one manifest of hashed inputs and code (FREEZE-1, FREEZE-2), ran once, and kept its outputs. The required-path answerer in the frozen runs is a deterministic extractive baseline, tuned on dev only, not a model. Questions and labels for the held-out splits stay private; IDs, counts, and outcomes are published.
FREEZE-1, split eval (24 questions, 2026-11-21): hold. Retrieval reached 20 of 20 required claims in every condition, and no gated answer held an unsupported, stale, wrong-scope, or outranked claim. Correct supported answers were 15 of 18, with no false abstentions. Every condition still failed two checks: it answered non-answerable questions (direct 3, BM25 3, semantic 2, hybrid 2, of 6), and correct abstentions were 3 of 6. Diagnosed after the run, and labeled as such: the extractive answerer returned true, correctly cited, in-effect claims about other rules, and the checker never asked whether a claim answers the question. Licensed is not the same as responsive.
FREEZE-2, split eval2 (24 new questions, 2026-11-21): hold. One system change: the ninth check, responsive, which requires the question to name at least one subject term of the supporting fact's field. Because eval had been seen, FREEZE-2 was scored once on a new split whose questions were written and hashed before the check existed in code. Every safety check passed in every condition: 0 unsupported, stale, wrong-scope, or outranked claims, 0 answers on 7 non-answerable questions, and correct abstentions 7 of 7. Retrieval was 19 of 19. But correct supported answers were 12 of 17 (0.71) in every condition, below 0.75. Three answerable questions (H01, H06, H15) lost a true, supported claim at the responsive check, among them one asked only by identifier and one asked in everyday words, and two two-passage questions (H19, H20) were incomplete because the extractive answerer returned one claim.
LIVE-001, Claude on a 12-question eval2 subset (2026-11-21): descriptive only. claude-sonnet-5-5 at effort medium, max_tokens 900, JSON outputs, uncached, 4 conditions by 3 repetitions in rotated order: 144 of 144 calls succeeded with no retries, an estimated USD 1.19 at that day's list prices of USD 2 and USD 10 per million input and output tokens (Claude pricing). Before the checker, live answers were correct on 24 of 24 answerable runs and abstained correctly on 12 of 12 non-answerable runs in every condition, including both two-passage questions the extractive baseline left incomplete. After the FREEZE-2 checker, 18 of 24 stayed correct: the responsive check dropped H01 and H06 in every repetition. Replayed offline through the FREEZE-1 checker, all 24 stayed. Every question had the same outcome in all three repetitions.

Extended description
Title: "Safer, then too cautious." Beside it, a dashed vertical mark labeled "the 0.75 bar", the decision rule's threshold for correct supported answers, which appears as a dashed line on the first bar of the first two panels. Three panels:
Panel 1, "FREEZE-1", "eval, extractive": "Correct supported": "15 of 18" (the bar ends past the dashed 0.75 line; the same in every condition); "Unsupported answers": "direct, BM25" "3 of 6"; "semantic, hybrid" "2 of 6"; "False abstentions": "0 of 18"; Verdict chip, amber: "hold: S4, Q4 failed".
Panel 2, "FREEZE-2", "eval2, extractive": "Correct supported": "12 of 17" (the bar ends short of the dashed 0.75 line); "Unsupported answers": "every condition" "0 of 7"; "False abstentions": "3 of 17"; Verdict chip, amber: "hold: Q2 failed".
Panel 3, "LIVE-001", "Claude, 12 questions", "3 runs per condition": "Correct supported": "raw, no checker" "24 of 24"; "FREEZE-2 checker" "18 of 24"; "FREEZE-1 checker" "24 of 24"; Verdict chip, blue: "descriptive only".
Footer, two lines: "Synthetic data. Simulated services." and "Live: claude-sonnet-5-5, effort medium. Recorded 2026-11-21."
What it shows: the first frozen checker let unsupported answers through on questions the corpus cannot answer. The second checker stopped every one of them, but it also refused 3 answerable questions and fell under the 0.75 bar, so both frozen runs are held. In the live run, Claude's raw answers were correct on all 24 answerable runs in every condition, and abstained correctly on all 12 non-answerable runs; the FREEZE-2 checker's responsive check dropped questions H01 and H06 in all three repetitions, and replaying the same answers through the FREEZE-1 checker kept all 24. LIVE-001 is one 12-question subset, so it describes these runs and decides nothing.
Two more observations from the live run, both disconfirming in their own way:
- Prose can drift from checked fields. In 28 of 123 live claims, the claim sentence did not contain the structured value the checker verified (a substring audit, not entailment). In a 20-claim sample judged by the AI author, with human review still pending, 19 cited spans supported their value and 1 partly did; 4 licensed claims had sentences that left the value out, 1 licensed claim's sentence added an assertion its span does not contain, and 4 responsive claims were dropped by the checker.
- Cost followed context. The direct condition sent 264,060 input tokens over its 36 calls against about 63,000 to 64,000 for each ranked condition, about three times the estimated cost (USD 0.60 against about USD 0.19 to 0.20), with similar median latency (2.1 to 2.2 seconds in every condition).
Decide with a rule written first
The rule was declared before any held-out run:
| Check | Metric (after the checker, eligible filter, k = 5) | Threshold |
|---|---|---|
| S1 | Unsupported decisive claims | 0 |
| S2 | Stale-as-current claims | 0 |
| S3 | Wrong-scope or outranked claims | 0 |
| S4 | Answers on non-answerable questions | 0 |
| Q1 | Claim recall at 5 | at least 0.90 |
| Q2 | Correct supported answers | at least 0.75 of answerable |
| Q3 | False abstentions | at most 0.20 of answerable |
| Q4 | Correct abstentions | at least 0.80 of non-answerable |
| Q5 | Mean context characters | at most 20,000 |
Among conditions that pass every check, choose the lowest complexity: direct, then BM25, then semantic, then hybrid. If none passes, hold, report each failed check with its question IDs, and name the evidence that would change the decision.
Applied to FREEZE-2: every condition passes S1 to S4, Q1, Q3 (3 of 17 is 0.18), Q4, and Q5, and fails Q2. The decision is hold. No retriever is recommended, and the measurements do not justify adding one.
What would change it, in order of evidence:
- A new freeze and a fresh held-out split.
eval2has now been seen, so any repair scored on it is exploratory. An exploratory rerun ofevalunder FREEZE-2 code gave 15 of 18 correct supported, 0 unsafe claims, and 6 of 6 correct abstentions; it is labeled exploratory and is not held-out performance. - A responsive check that knows identifiers and everyday synonyms for a field, tested against the questions it wrongly dropped.
- An answerer that returns every required claim for multi-part questions.
- Claim sentences rendered from checked fields, so what users read is what was verified.
If the answer stage clears the bar on fresh questions with the safety checks still at zero, the rule would choose direct, the simplest condition, because it ties the ranked retrievers on every quality count here and stays within the context budget. The live run shows its price: about three times the input cost per call on this small library, rising with the library's size. A ranked retriever earns its complexity when the eligible library outgrows the context budget, or when cost or latency at your volume says so. Measure that on your own corpus; this guide's numbers cannot settle it for you.
A pass under this rule would be evidence on 24 synthetic questions over 21 documents. It would not be a production guarantee: real questions, corpora, scopes, and answerers differ.
Choosing lexical, hybrid, or direct
This is guidance from the mechanisms above, not a measured ranking:
- Direct when the eligible set for a request fits comfortably in your context and cost budget. It removes ranking as a failure mode; the eligibility filter and the checker still apply.
- BM25 when questions carry exact identifiers, codes, or names, with a tokenizer that keeps them whole. Check it on paraphrases before relying on it.
- Semantic when questions and documents use different words for the same rule. It cannot embed new text without an encoder at query time.
- Hybrid when you see both kinds of question and have measured that neither alone reaches the claims; it pays for two indexes and a larger candidate union.
When not to use RAG
- The answer lives in a structured system of record. Query the record (a ledger, an HR system, a pricing table) and cite the row; retrieval over prose copies of it adds a way to be stale.
- Nobody owns the revisions. If superseded and draft documents are not marked, every filter in this guide has nothing to filter on, and the assistant will cite stale text with confidence. Fix the content metadata first.
- A wrong answer is costly and claims cannot be checked. Without stored facts or a reviewed sample to check support against, route the question to a person rather than shipping fluent unverified answers.
- The library is small and stable. Read it directly, as the
directcondition does, and spend the effort on the checker.
Setup and runtime troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Unsupported engine on install | Node.js older than 22.18 | Install Node.js 22.18 or newer and install again |
ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION | A pinned package is less than a day old; pnpm 11 and later wait 1440 minutes by default (pnpm 11.0 release notes) | Wait until a day after the publish time the error names; do not turn the check off |
A FAIL line in pnpm preflight about data/MANIFEST.json | A shipped input, test, or grader file was edited | Restore it from the ZIP; edit only the files each lab names |
Set ANTHROPIC_API_KEY in your shell to use --live | pnpm live --live without a key | Not needed for any lab; nothing was sent |
Operational boundaries
- What these results are: two deterministic held-out runs and one descriptive live run on synthetic data, each on 24 questions or fewer, on one day. They show how the method finds problems; they are not production accuracy.
- What the checker can prove: that each kept claim cites existing, delivered, in-effect, in-scope text containing a stored fact with the claimed value, about what the question asks, from the governing authority. It cannot prove that the register of facts is right, that a question was understood, or that the claim sentence says what was checked (
F-14). - Where the facts come from: the kit's register of facts is authored with the corpus. In your system it is the expensive part: structured fields extracted and reviewed for each policy revision. Without it, support falls back to weaker evidence such as substring match or a model judge, which this guide does not treat as proof.
- Labels: the held-out labels are the author's, checked by an automated blinded second pass (48 of 48 classes and 43 of 43 required claims agreed for the first two splits; 24 of 24 and 19 of 19 for
eval2). An automated second pass is not a human; human review is pending. - Live calls: optional, opt-in, capped by
--max-usd, uncached, and billed to your own account. Never put real company documents in the synthetic kit. - Not covered: PageIndex and other tree-search retrieval, rerankers, and the full contextual retrieval pipeline; none was run here.
Quick reference
Rules
- Filter by status, half-open effective interval, and scope before ranking, in one place, for every mode.
- Keep identifiers whole in the tokenizer, and keep digits.
- Score retrieval at claim grain, separately from answers and from abstentions.
- Cite a revision, a passage, and an end-exclusive range; resolve model quotes to ranges in your own code.
- Check support against stored facts, then responsiveness, then authority against every eligible source.
- Abstain with the most specific reason and a next step; count false abstentions too.
- Write the decision rule before the held-out run; when nothing passes, hold.
Commands
| Command | What it does |
|---|---|
pnpm preflight | Confirms Node.js, the corpus, vectors, and the integrity manifest |
pnpm rank D01 --mode bm25 | Ranked passages for one dev question (--mode bm25, semantic, hybrid, or direct; --filter eligible, date-only, or none) |
pnpm inspect D09 | One question under BM25, semantic, and hybrid, with the rank of acceptable evidence |
pnpm compare | Dev retrieval scorecard (--stale adds unfiltered runs) |
pnpm answer --mode hybrid | Extractive answers, your checker, and the answer and abstention scorecards |
pnpm check:fixtures | Your checker on the eleven LAB-003 fixtures |
pnpm scorecard --frozen | The frozen held-out scorecard (FREEZE-2, with FREEZE-1's decision) |
pnpm gallery F-01 | One failure card |
pnpm lab1:check, pnpm lab2:check, pnpm lab3:check, pnpm test | Acceptance checks |
pnpm reset | Removes run outputs and confirms the shipped inputs are untouched (extract the ZIP again to undo code edits) |
05Hands-on lab
Lab 1: Retrieve the right revision
Employees ask the policy assistant about stipends, record retention, and sign-in rules. The library keeps every revision, including superseded and draft ones, and one policy family has separate US and EU versions. Retrieval has three seeded faults: the eligibility filter is computed and then ignored, the first day of a revision is treated as outside it, and the tokenizer drops digits, so RC-14/B turns into the common fragment rc. One filter for every mode. Do not filter in BM25 only, and do not change scores to push the right passage up. Every output below was recorded offline on 2026-11-21 (UTC) with Node.js 22.23.5 and pnpm 12.21.0 at $0.00 (Synthetic data. Simulated services.); pnpm's command echo, [ELIFECYCLE] lines, blank lines and stack frames are left out, and your paths start where you unpacked retrieval-labs.zip.
Acceptance checks
- pnpm lab1:check passes 6 of 6 (exit code 0).
- In pnpm compare, every condition shows the same pool (49.7 eligible passages on average, not 88).
- You can say in one sentence why D02 (asked on 2026-07-01, the first day of EQP-100 r2) failed before step 4.
Observe the failing acceptance tests
$ pnpm lab1:checkInitial: Starter code, exit code 1; assertion detail lines left out
not ok 1 - first day of a revision is inside its interval; the end date is outside error: |- EQP-100 r2 starts 2026-07-01 and must be in effect that day ok 2 - drafts and out-of-scope documents are never eligible not ok 3 - every retrieval mode ranks only the eligible pool error: |- D02 bm25: pool should hold only eligible passages not ok 4 - the first-day question retrieves the new revision, not the old one error: 'D02 (as_of 2026-07-01) should deliver EQP-100@r2#s2.p1; got FAQ-EQP@r1#s2.p1, EQP-100@r1#s3.p1, EQP-100@r1#s2.p1, EQP-100@r2#s1.p1, FLT-010@r1#s3.p2' ok 5 - history is kept: a past date still retrieves the revision in effect then not ok 6 - an exact identifier ranks its own passage first under BM25 error: |- D07 asks about RC-14/B # tests 6 # pass 2 # fail 4Final: After steps 3 to 5, exit code 0
# tests 6 # pass 6 # fail 0Trace one question: the filter label is not the filter
$ pnpm rank D01 --mode bm25D01 as_of 2026-11-01 applies_to worker=employee Q: How much can an employee claim under the home office stipend? mode bm25 filter eligible k 5 depth 20 pool 88 passages context 1526 chars 1 6.7912 FAQ-EQP@r1#s2.p1 guidance in effect "How much is the home office stipend? The home office stipend is up to USD 500, paid once." 2 6.0801 EQP-100@r1#s2.p1 policy superseded (ended 2026-07-01) "Employees may claim a one-time home office stipend of up to USD 500 for a desk or chair." 3 5.6716 EQP-100@r1#s3.p1 policy superseded (ended 2026-07-01) "Submit the claim within 60 days of purchase."Apply the shared filter, make the interval half-open, and keep identifiers whole
$ pnpm rank D07 --mode bm25Initial: Starter code; the first two hits
D07 as_of 2026-11-01 applies_to none Q: How long must RC-14/B records be kept? mode bm25 filter eligible k 5 depth 20 pool 88 passages context 1512 chars 1 2.5557 REC-SCHED@r6#s1.p1 standard superseded (ended 2026-01-01) "This schedule sets how long Example Instruments keeps each record class. Unless a class says otherwise, ret..." 2 2.4823 REC-SCHED@r7#s1.p1 standard in effectFinal: After steps 3 to 5; the first hit
mode bm25 filter eligible k 5 depth 20 pool 61 passages context 1414 chars 1 4.8835 REC-SCHED@r7#s2.p3 standard in effect "RC-14/B, vendor invoices: retain for 7 years after the fiscal year in which the invoice was paid."Compare the four conditions on the dev split
$ pnpm compareInitial: Starter code; the summary rows
condition recall@k (claims) full coverage@k multi-passage@k MRR@depth doc recall@k doc MRR disq. delivered pool delivered union context chars bm25/eligible 17/23 (0.74) 13/19 (0.68) 2/2 (1.00) 0.584 21/23 (0.91) 0.677 16 88.0 5.0 N/A 1486 semantic/eligible 21/23 (0.91) 17/19 (0.89) 1/2 (0.50) 0.763 22/23 (0.96) 0.771 22 88.0 5.0 N/A 1459 hybrid/eligible 20/23 (0.87) 16/19 (0.84) 2/2 (1.00) 0.652 22/23 (0.96) 0.716 19 88.0 5.0 24.3 1454 direct/eligible 23/23 (1.00) 19/19 (1.00) 2/2 (1.00) N/A 23/23 (1.00) N/A 23 88.0 88.0 N/A 25928Final: After steps 3 to 5; the summary rows
condition recall@k (claims) full coverage@k multi-passage@k MRR@depth doc recall@k doc MRR disq. delivered pool delivered union context chars bm25/eligible 22/23 (0.96) 18/19 (0.95) 2/2 (1.00) 0.787 23/23 (1.00) 0.860 4 49.7 4.8 N/A 1397 semantic/eligible 22/23 (0.96) 18/19 (0.95) 1/2 (0.50) 0.921 23/23 (1.00) 0.947 4 49.7 5.0 N/A 1468 hybrid/eligible 23/23 (1.00) 19/19 (1.00) 2/2 (1.00) 0.829 23/23 (1.00) 0.895 4 49.7 5.0 21.5 1475 direct/eligible 23/23 (1.00) 19/19 (1.00) 2/2 (1.00) N/A 23/23 (1.00) N/A 4 49.7 49.7 N/A 14518Show the solution discussion
Run "node ../solutions/apply-solution.mjs lab1 /tmp/retrieval-lab1-solution" and diff the three files against yours.
More labs in this guide
Lab 2: Compare retrieval and decide
The team wants to know whether to ship BM25, semantic, hybrid, or a direct read of every eligible document, and the safety owner wants a predeclared rule rather than a favorite. This lab changes no code. You read the dev scorecard from your repaired retriever, then the author's frozen evaluation scorecard, and write a recommendation that a script checks for evidence and bounds, not for agreement. Every output below was recorded offline on 2026-11-21 (UTC) with Node.js 22.23.5 and pnpm 12.21.0 at $0.00 (Synthetic data. Simulated services.); pnpm's command echo, [ELIFECYCLE] lines, blank lines and stack frames are left out, and your paths start where you unpacked retrieval-labs.zip.
Lab 3: Build the checker
An answer is a list of atomic claims, each with references to a document revision, passage, and character range, or an abstention with a reason and a next step. The starter checker compares a quote with the whole passage and checks nothing else, so it licenses an exact quotation from a superseded revision and a valid pointer to an unrelated sentence. You build the checks that decide which claims the evidence licenses. data/fixtures/lab3.json holds eleven scripted answer doubles, L3-A to L3-K. They are constructed test inputs, not model output. Every output below was recorded offline on 2026-11-21 (UTC) with Node.js 22.23.5 and pnpm 12.21.0 at $0.00 (Offline: scripted model double, not Claude. Synthetic data. Simulated services.); pnpm's command echo, [ELIFECYCLE] lines, blank lines and stack frames are left out, and your paths start where you unpacked retrieval-labs.zip.
06Failure gallery
Fourteen ways this breaks, and how you will know
Twelve of these reproduce in the lab; the other two were observed in this guide's recorded runs. Learn the symptom first; it is what you will actually see.
- F-01
gallery F-01ReproducibleIdentifier split into common fragments
- Symptom
- D07 asks about RC-14/B; the right passage ranks sixth under BM25.
- Diagnosis
- The starter tokenizer turns SEC-AC-07 RC-14/B into ["sec","ac","rc"], common fragments that match many passages.
- Fix
- Keep identifiers whole, also emit their parts, and keep digits (LAB-001 step 5).
- Verify
- "pnpm rank D07 --mode bm25" shows REC-SCHED@r7#s2.p3 at rank 1; pnpm lab1:check.
- F-02
gallery F-02ReproducibleSimilar policy for the wrong scope
- Symptom
- D15 asks about EU employees; with the scope filter off, semantic retrieval ranks the US travel policy's receipt rule second, and the constructed answer L3-G cites it.
- Diagnosis
- Near-duplicate text across regions; similarity cannot tell scope.
- Fix
- Filter by scope before ranking and check scope in the checker.
- Verify
- "pnpm rank D15 --mode semantic"; pnpm lab3:check (L3-G).
- F-03
gallery F-03ReproducibleSuperseded or future revision treated as current
- Symptom
- With no eligibility filter, BM25 puts superseded EQP-100 r1 second for D01 and the in-effect r2 tenth, and the extractive answer says USD 500 within 60 days: stale as current.
- Diagnosis
- The old revision's wording is closer to the question than the current one's.
- Fix
- Apply the shared date filter to every mode; check effective dates in the checker.
- Verify
- "pnpm rank D01 --mode bm25 --filter none"; pnpm lab1:check; pnpm lab3:check (L3-C).
- F-04
gallery F-04ReproducibleSecond passage missing from the top k
- Symptom
- D13 needs three claims from three passages; under semantic retrieval one ranks sixth, and the answer is incomplete.
- Diagnosis
- Top-1 or document-level metrics hide a missing second passage.
- Fix
- Measure claim-grain and multi-passage coverage; compare conditions at the same k.
- Verify
- "pnpm inspect D13"; pnpm compare.
- F-05
gallery F-05ReproducibleInvalid span: fabricated range or rewritten quote
- Symptom
- A reference whose range runs past the end of its passage (L3-D), or an in-range reference whose quote was reworded (L3-E).
- Diagnosis
- The reference looks valid until its bounds and text are recomputed.
- Fix
- Check range bounds and that the quote equals the cited slice.
- Verify
- Pnpm check:fixtures; pnpm lab3:check (L3-D, L3-E).
- F-06
gallery F-06ReproducibleValid pointer that does not support the claim
- Symptom
- L3-B cites a real, in-effect passage that says 90 days, for a claim of USD 600; L3-I keeps the value and drops its condition ("unless the sponsoring manager renews").
- Diagnosis
- A resolvable citation proves existence, not support.
- Fix
- Require a register fact inside the cited range that states the value, unit, qualifiers, and subject.
- Verify
- Pnpm lab3:check (L3-B, L3-I).
- F-07
gallery F-07ReproducibleOverconfident answer to a missing rule
- Symptom
- D21 asks about pet relocation, which no policy covers; the raw answer offers a lodging cap and a rule about where mileage is measured from.
- Diagnosis
- An answerer produces plausible claims from whatever was delivered.
- Fix
- Block claims without support or responsiveness and abstain with no_eligible_support and a next step.
- Verify
- Pnpm lab3:check (L3-F); "pnpm answer --mode bm25".
- F-08
gallery F-08ReproducibleUnnecessary abstention
- Symptom
- A checker that compares each quote with the whole passage rejects the correct sentence quote in L3-A, so a supported answer becomes an abstention.
- Diagnosis
- Quote fidelity means the quote equals the cited slice, not the passage. In D04, the direct-read answer adds an unrelated travel claim; the gate drops that claim and keeps the supported 90-day one.
- Fix
- Compare quotes with the cited slice, drop the failing claim rather than the answer, and keep one decisive claim per part of the question, because a conflict or scope ambiguity on any claim abstains the whole answer.
- Verify
- Pnpm check:fixtures (L3-A); "pnpm answer --mode direct".
- F-09
gallery F-09ReproducibleNew question cannot be embedded offline
- Symptom
- Semantic mode reports that no frozen vector exists for a new question.
- Diagnosis
- The offline kit ships vectors for the corpus and the dev questions only.
- Fix
- Use BM25 for new text, or provision tools/encoder to generate vectors.
- Verify
- Pnpm rank --text "How much can I spend on a standing desk?" --mode semantic shows the message; the same command with --mode bm25 ranks.
- F-10
gallery F-10ReproducibleLower-authority guidance cited over the policy
- Symptom
- For D01, semantic retrieval ranks the FAQ first and the extractive answer cites its USD 500.
- Diagnosis
- The FAQ is eligible; only authority says the policy governs.
- Fix
- Check authority against every eligible source for the same rule, not only the delivered ones.
- Verify
- "pnpm answer --mode semantic" with the worked solution reports 0 wrong-scope or outranked claims after the checker.
- F-11
gallery F-11ReproducibleUnresolved conflict answered with one side
- Symptom
- D23: two in-effect policies give USD 0.70 and USD 0.67 per mile; the raw answer picks one.
- Diagnosis
- Same-authority sources disagree, and nothing in the order of authority settles it.
- Fix
- Compare the supporting fact with every same-rule fact at the same authority; abstain with conflict_unresolved.
- Verify
- "pnpm answer --mode hybrid".
- F-12
gallery F-12ReproducibleTrue claim that answers a different question
- Symptom
- For D01, the raw BM25 answer adds a true, cited, in-effect claim about where mileage is measured from.
- Diagnosis
- Licensed is not responsive; this is the FREEZE-1 failure in miniature.
- Fix
- After support, check that the supporting fact is about what the question asks (isResponsive in src/check/support.ts); drop it otherwise, and abstain when nothing responsive is left.
- Verify
- Pnpm lab3:check (L3-K); "pnpm answer --mode bm25".
- F-13Observed
The responsive check abstains on answerable questions
- Symptom
- H01, H06, and H15 end as false abstentions in every condition, and the live run loses H01 and H06 in every repetition.
- Diagnosis
- The check needs a subject term from the field registry in the question; a question asked by identifier only, or in everyday words, names none.
- Fix
- Not yet tested: teach the check identifiers and synonyms per field, then score it on a new freeze and a fresh split.
- Verify
- False abstentions at most 0.20 and correct supported at least 0.75 on the new split, with every safety check at 0.
- F-14Observed
Claim sentence omits the checked value
- Symptom
- 28 of 123 live claims had sentences that did not contain their structured value; 4 such claims were licensed in a 20-claim sample.
- Diagnosis
- The checker verifies value and qualifiers; nothing required the sentence to match.
- Fix
- Not yet tested: render the displayed sentence from checked fields, or add a sentence check, under a new freeze.
- Verify
- A prose audit count of 0 on a new live run.
07Assessment
Transfer task and rubric
Use your repaired kit on an unseen synthetic library: deliver eligible evidence, answer with checkable claims or abstain, and decide from a scorecard by the rule.
| Criterion | 0 · Emerging | 1 · Developing | 2 · Proficient | 3 · Exemplary |
|---|---|---|---|---|
| C1 Eligible evidence and rejections (OBJ-001) | Fewer than 9 of 13 questions have correct evidence | At least 9 of 13 questions have correct evidence, but two or more questions miss it or two or more near misses are unrejected or carry the wrong reason | At least 12 of 13 questions have correct, eligible, delivered evidence, and at most one near miss is unrejected or mislabeled | Proficient, and every question's evidence and every near-miss rejection is correct, with the right reason |
| C2 Retriever comparison (OBJ-002) | Fewer than 3 of the 6 reported ranks are correct | 3 or 4 of the 6 ranks are correct | 5 of the 6 ranks are correct | Proficient, and all 6 ranks are correct |
| C3 Checkable answers (OBJ-003) | Fewer than 6 of the 9 answerable questions have a correct, fully supported answer | 6 or 7 of the 9 are correct and supported | 8 of the 9 are correct and supported, and every kept claim passes the checker | Proficient, and all 9 are correct, including every part of a multi-part question |
| C4 Abstentions (OBJ-004) | Abstains on fewer than 3 of the 4 non-answerable questions | Abstains on 3 of the 4, or on all 4 with two or more wrong reasons or more than one false abstention | Abstains on all 4, at least 3 with the most specific reason, and at most one false abstention | Proficient, and all 4 reasons are the most specific one with no false abstention |
| C5 Decision from the rule (OBJ-005) | No decision, or a decision the rule does not support on this scorecard | A decision the rule supports, but without the decisive check and its count, or without the safety rule, or mixing retrieval results with answer verdicts | A decision the rule supports, naming the decisive check with its count, at least three scorecard counts, the safety rule, and retrieval kept apart from answer verdicts | Proficient, and names the question behind the decisive check and the counterfactual: which condition the rule would choose if that question changed |
| C6 Limits and next evidence (OBJ-005) | States none of: synthetic data, small denominators, or what would change the decision | States some of these limits, but not both that the case is synthetic and what evidence would change the decision | States that the case is synthetic and what evidence would change the decision, and either the small denominators or the specific question that drives the result | Proficient, and states all four: synthetic, small denominators, the specific question, and the next evidence |
Proficiency standard: Proficient or higher on every criterion. Proposed local standard, not an official certification.
The full brief
The assessment hands you an unseen synthetic library, Example Harbor Ferries: 9 documents, 26 passages, and 13 questions with their own dates and scopes, some of which the library cannot answer. With your repaired kit, you deliver eligible evidence and name the near misses you rejected (ASM-001), write checkable claims or abstain with the most specific reason (ASM-002), and then apply the decision rule to a scorecard of the reference pipeline on that case and bound the conclusion (ASM-003).
Two files in work/asm/: submission.json (Parts 1 and 2) and decision.md (Part 3). The JSON format, with a worked dev example, is in the lab handout in the download, under "Transfer assessment submission format". Submit Parts 1 and 2 first; your instructor then gives you the transfer scorecard for Part 3.
Before you submit Parts 1 and 2, run node scripts/check-answers.ts --case ../asm-case work/asm/submission.json and fix every FIX line. The self-check uses only your code and never reads labels. A clean result means every reference resolves and your checker licenses every claim you kept. It does not mean an answer is correct.
The proposed local standard is Proficient or higher on every criterion and no critical failure: no unsupported claim kept, no stale, out-of-scope, or outranked rule treated as current, no answer to a non-answerable question, and no copied instructor-only text. It has been calibrated on 13 constructed submissions, not on learners.
08Instructor kit
Teach it to a room
Facilitation guide and lab solutions
A 160-minute session plan across LAB-001 to LAB-003, with the three-part assessment as a separate 60-minute session, hint ladders and recovery for each checkpoint, and lab solutions whose offline check script runs every checkpoint and three planted mistakes at $0.00.
Delivery for 12-person and 100-person rooms
A one-room plan for about 12 learners working LAB-003 in pairs, and a one-day plan for about 100 learners with breakouts, a help queue and fixed progress checks. These are scenario designs; no cohort has run them yet.
Dry run and observation rubric
A proposed train-the-trainer sequence and a 50-minute observed dry run: teach-back of the framing, one lab walk-through, and a failure-recovery drill, scored with a written observation rubric. No dry run has been observed yet.
Clearance record
A per-instructor, per-version clearance form filled in only from an observed dry run and signed by an authorized person. Blank: nobody has been cleared to teach this guide.
Evaluation plan and learner report template
Evidence to collect per delivery, blind double scoring before any high-stakes use, data handling rules, an issue log, and a monthly learner report template with numerators and denominators. Blank until piloted.
10Sources and verification
- [1]Anthropic Academy, Building with the Claude API · anthropic.skilljar.comchecked Nov 21, 2026
- [2]Anthropic, Introducing Contextual Retrieval (published 2024-09-19) · anthropic.comchecked Nov 21, 2026
- [3]Claude API docs, Citations · platform.claude.comchecked Nov 21, 2026
- [4]Claude API docs, Embeddings · platform.claude.comchecked Nov 21, 2026
- [5]Claude API docs, Search results · platform.claude.comchecked Nov 21, 2026
- [6]Claude API docs, Structured outputs · platform.claude.comchecked Nov 21, 2026
- [7]Claude API docs, Effort · platform.claude.comchecked Nov 21, 2026
- [8]Claude API docs, Models overview · platform.claude.comchecked Nov 21, 2026
- [9]Claude API docs, Pricing · platform.claude.comchecked Nov 21, 2026
- [10]@anthropic-ai/sdk changelog (1.12.0, 2026-11-20) · github.comchecked Nov 21, 2026
- [11]Node.js release index (v22.23.5, 2026-11-11) · nodejs.orgchecked Nov 21, 2026
- [12]Node.js 22 docs, Modules: TypeScript · nodejs.orgchecked Nov 21, 2026
- [13]npm registry, pnpm (12.21.0, 2026-11-13) · registry.npmjs.orgchecked Nov 21, 2026
- [14]npm registry, @huggingface/transformers (4.3.2, 2026-10-27) · registry.npmjs.orgchecked Nov 21, 2026
- [15]BAAI, bge-small-en-v1.5 model card (revision 5c38ec7) · huggingface.cochecked Nov 21, 2026
- [16]Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) · staff.city.ac.ukchecked Nov 21, 2026
- [17]Apache Lucene 10.5.0, BM25Similarity · lucene.apache.orgchecked Nov 21, 2026
- [18]Cormack, Clarke, and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (2009) · cormack.uwaterloo.cachecked Nov 21, 2026
- [19]Dev Guide: Agent Evaluations That Measure Real Outcomes (version 1.1.0) · jordanhudgens.comchecked Nov 21, 2026
- [20]Dev Guide: Context Engineering for Long-Running Agents (version 1.0.0) · jordanhudgens.comchecked Nov 21, 2026
- [21]University of Hawaii at Manoa, Creating and Using Rubrics · manoa.hawaii.educhecked Nov 21, 2026
- [22]Carnegie Mellon University Eberly Center, Creating and Using Rubrics · cmu.educhecked Nov 21, 2026
- [23]pnpm settings, dependency resolution · pnpm.iochecked Nov 21, 2026
- [24]pnpm 11.0 release notes · pnpm.iochecked Nov 21, 2026
Independent work. Not affiliated with or endorsed by any vendor named in this guide.
