Skip to content
Jordan Hudgens
Illustration: a small white capsule robot fits a glowing mint segment into a loop of track on a dark workbench, beside a cracked amber segment and an empty tray.AI-generated illustration
Guide 01IntermediateAgents and tool useEvaluation and testing

Build, Break, and Repair a Reliable Tool-Using Agent

Build a bounded agent loop in TypeScript, break it with seeded faults, and repair it so the system of record, not the model's message, decides success.

Reading
40 min
Workshop
90 min
Stack
TS · Node.js 22 · Anthropic SDK 0.133.0
Version
1.2.0
Last verified
Oct 13, 2026

What you will build and be able to do

Each outcome has an observable criterion. The lab and the assessment check exactly these, nothing more and nothing less.

  1. OBJ-001

    Build a bounded tool-use loop

    pnpm test:lab1 passes 5 of 5, including the two-tools-in-one-turn and step-cap tests.

  2. OBJ-002

    Trace every call to its result

    For a recorded run, you name the tool_use ids involved and the verifier line that proves the record state.

  3. OBJ-003

    Diagnose a seeded failure

    Each diagnosis names a harness behavior as the cause, with trace and verifier evidence.

  4. OBJ-004

    Repair the harness at the tool boundary

    pnpm test:lab3 passes 7 of 7, and every fault ends in verified success or a safe escalation.

  5. OBJ-005

    Decide success from the record

    You explain, citing a check from your own pnpm verify output, why the record and not the message decides success.

Runs offline in minutes

  • Comfortable reading and writing one programming language (the lab is TypeScript; JavaScript, Python, or Go readers can follow it)
  • JSON and basic command-line use
  • Has made at least one HTTP API callPrimer in this guide
  • Node.js 22.18 or later and pnpm 12; an Anthropic API key is optional, because every lab step also runs offline
Terminalbash
corepack enable pnpm          # once per machine with Node.js 22cd labs/starterpnpm installpnpm preflightpnpm reset2 tickets, 2 orders, 3 policies, 0 payouts, 0 approvals
Lab starter and reference solutions · ZIP · tested at v1.2.0

The harness owns the loop

The model proposes calls; everything that touches the world, including deciding success, is code you own.
Diagram: the model sends tool_use requests to your harness and receives tool_result replies with the same id. Inside the harness are a bounded loop, a tool boundary that calls services and asks an operator to approve writes, and a stop-and-hand-off step. A verifier reads the system of record and reports one of three outcomes.
Extended description

Title: "The harness owns the loop." Subtitle: "The model proposes. Your code decides what runs, when to stop, and what counts as done."

Left column, two boxes:

"Model": "Claude, or the offline scripted double."

"Operator" (raised-hand icon): "approves one exact write."

Center, a dashed blue region labeled "Your harness" containing three stacked boxes:

1. "Bounded loop" (loop icon): "one tool_result per tool_use, in one message"; "MAX_STEPS = 10; every stop reason is named."

2. "Tool boundary" (shield icon): "validate every result before the model sees it"; "writes: policy check, approval, idempotency key."

3. "Stop and hand off" (wrench icon): "repeated-failure breaker after 2 identical errors"; "every non-success stop sets needs_human."

Right column:

"Services": "policy API, payments API (simulated, fault injection)."

A database cylinder, "System of record": "tickets, orders, payouts, approvals", and the file name .state/desk.json.

"Verifier" (ledger icon, mint border): "reads the record, never the transcript." Below it, the three possible outcomes: verified_success, safe_escalation, failed.

Arrows and their labels:

Model to Bounded loop, white: "tool_use (id)".

Bounded loop back to Model, mint: "tool_result (same id)".

Tool boundary to Operator, white: "approve?".

Tool boundary to Services, white: "calls".

Services down to System of record: "writes".

Stop and hand off to System of record, dashed amber: "escalates".

System of record down to Verifier, mint: "reads".

Footer: "Example Outfitters is synthetic. Component names match the lab code."

The point of the figure: the model only proposes calls and receives results. Everything that touches the world (calling services, asking a human, stopping, judging success) sits in code the learner owns.

Why tool-using agents fail quietly

A tool-using agent is a model in a loop. On each turn the model either answers or asks your code to run a tool, such as "look up order ORD-5521" or "refund $48.00." Your code runs the tool, sends the result back, and asks again. The model never touches your systems directly. Everything it changes, it changes by asking code you wrote (tool use overview).

That code is the harness: the loop, the tool handlers, and the rules around them. This guide is about the harness, because that is where reliability lives. Anthropic's engineering team describes tools as "a contract between deterministic systems and non-deterministic agents" (Writing effective tools for agents). The model is the non-deterministic side. The harness is the deterministic side, and it is the only side you can make guarantees about.

A harness has four jobs:

  1. Run the loop correctly. Every tool request gets exactly one result, in the shape the API requires.
  2. Enforce the boundary. Check what tools return before the model relies on it, and refuse unsafe writes in code, whatever the model asks for.
  3. Decide when to stop. The model may never stop on its own. The harness must.
  4. Verify the outcome. Decide success from the state of the system, not from the model's last sentence.

Agents fail quietly because the conversation stays fluent while the world goes wrong. Two terms from agent evaluation make the distinction precise. The transcript is the record of what the agent said and did. The outcome is the final state of the environment. In Anthropic's example, a flight-booking agent might say "Your flight has been booked," but the outcome is whether a reservation exists in the database (Demystifying evals for AI agents). Keep that distinction in mind. By the end of the guide you will have seen an agent say "Refunded $48.00" while the ledger shows $96.00 paid.

Prerequisite bridge: one tool call, end to end

If you have built a tool loop before, skip to Set up the lab.

You describe each tool with a name, a description, and a JSON Schema for its input (define tools):

Tool definition: get_orderts
{  name: 'get_order',  description: 'Read one order by ID: line items with prices in cents, promised and actual delivery dates, and the amount already refunded. Read-only.',  strict: true,  input_schema: {    type: 'object',    properties: { order_id: { type: 'string', description: 'Order ID, for example ORD-5521.' } },    required: ['order_id'],    additionalProperties: false,  },}

strict: true asks the API to guarantee that the model's tool inputs match the schema; it requires additionalProperties: false (strict tool use). It constrains what the model sends you. It says nothing about what your tools send back.

When the model wants a tool, the response has stop_reason: "tool_use" and one or more tool_use blocks, each with its own id. This is step 2 of a real run in this lab, where Claude asked for two tools in the same turn:

Assistant turn, step 2 of a real runjson
{  "step": 2,  "stop_reason": "tool_use",  "tool_uses": [    { "id": "toolu_01AekwLPa3mAAE86DpUGfUfU", "name": "get_order", "input": { "order_id": "ORD-5521" } },    { "id": "toolu_01BXc3SMdsZe4dcbdbEXtdwU", "name": "search_policy", "input": { "category": "damaged_on_arrival" } }  ]}

Observed in run LIVE-04 (2026-10-13), from the lab's action log.

Your code runs both tools and replies with a user message that contains one tool_result per tool_use, each pointing back at its id with tool_use_id. If a tool failed, you still send a result, with is_error: true and a message that explains what went wrong (handle tool calls):

Tool results, one user messagejson
{  "role": "user",  "content": [    { "type": "tool_result", "tool_use_id": "toolu_01AekwLPa3mAAE86DpUGfUfU", "content": "{\"id\":\"ORD-5521\", ...}" },    { "type": "tool_result", "tool_use_id": "toolu_01BXc3SMdsZe4dcbdbEXtdwU", "content": "Policy service returned HTTP 503", "is_error": true }  ]}

The model continues until it answers with stop_reason: "end_turn" or something else stops it (how tool use works). That cycle is the whole mechanism. The rest of this guide is about running it safely.

Set up the lab

The lab project is support-desk-agent, in labs/starter/. It is TypeScript that Node.js runs directly: since Node.js 22.18.0, type stripping is enabled by default, so .ts files run without a build step (Node.js TypeScript docs). Node strips types but does not check them, so the project includes pnpm typecheck for that (erasableSyntaxOnly).

Terminalbash
corepack enable pnpm          # once per machine with Node.js 22; or: npm install --global pnpm@12.11.2cd labs/starterpnpm installpnpm preflightpnpm reset

Observed pnpm preflight output on the test machine (run 2026-10-17, with a Claude API key configured):

pnpm preflight (2026-10-17)text
PASS  Node.js: 22.23.3 (needs 22.18 or later for built-in TypeScript)PASS  pnpm: pnpm/12.11.2PASS  dependencies: run "pnpm install" if this failsPASS  writable project folder: /tmp/support-desk-qa-5nMT/starterPASS  live mode: available (model claude-sonnet-5-5) Ready.

On your machine, the folder path differs, and "live mode" reads not configured until you export a key. That is fine: it is reported as INFO, not FAIL.

The synthetic business

Example Outfitters is a synthetic retailer. Its system of record is one JSON file, .state/desk.json, restored from data/seed.json by pnpm reset. The agent's first job is ticket T-1042:

The order, ORD-5521, is one $249.00 tent delivered October 7. Policy POL-DOA-30 allows a refund for a damaged component of up to $60.00, reported within 30 days. The correct outcome is one human-approved refund of $48.00 and a resolved ticket.

Two models, one interface

The loop talks to anything that implements ModelClient:

  • The scripted double (--model scripted, the default) is offline test code, not Claude. It is cooperative but overconfident: it believes any result that looks like success, fills missing facts with the most generous guess, repeats failed calls forever, and always claims success. Every acceptance test runs against it, because your harness's guarantees have to hold when the model makes mistakes.
  • Claude (--model claude) runs live through the official SDK. Set ANTHROPIC_API_KEY in your shell. The default model is claude-sonnet-5-5 with effort medium; override them with AGENT_MODEL and AGENT_EFFORT (models overview, effort). Sonnet 5.5's documented default effort is high; this lab lowers it to keep classroom runs fast. For harder tasks, the platform's general advice is to start with Opus 5.5.

Each live happy-path run of the lab in our tests used between about 7,900 and 10,700 input tokens and fewer than 1,000 output tokens, about $0.02 to $0.03 at Sonnet 5.5's list price of $2 and $10 per million input and output tokens (pricing). Your numbers will vary.

Build the bounded loop

The decision. A loop that runs until the model says stop is a loop someone else controls. Make it bounded by construction, make every tool request get exactly one answer, and treat every way of stopping as a stop, not as success. Anthropic's agent guidance says the same in general terms: agents get ground truth from tool results at each step, and stopping conditions such as a maximum number of iterations keep you in control (Building effective agents).

The mechanism. Open src/agent.ts. The starter gives you the model call, the logging of each call, and a for loop capped at MAX_STEPS = 10. You write the body. This is the reference implementation of the part you write, from labs/solutions/checkpoint-a/src/agent.ts:

src/agent.ts (checkpoint A)ts
// LAB-001.1 Keep the assistant turn exactly as received, including any thinking blocks.messages.push({ role: 'assistant', content: response.content }); // LAB-001.2 The model says it is done. That ends the loop; it does not prove anything.if (response.stop_reason === 'end_turn') {  return { stop: 'end_turn', steps: step, finalText: text, messages };} // LAB-001.3 Any other stop reason is a stop, never an implicit success.if (response.stop_reason !== 'tool_use') {  const stop: AgentStop =    response.stop_reason === 'max_tokens' ? 'max_tokens' : response.stop_reason === 'refusal' ? 'refusal' : 'unexpected_stop';  return { stop, steps: step, finalText: text, messages };} // LAB-001.4 and .5 Run every requested tool, in order, and log each result by its id.// Sequential on purpose: two writes in one turn must not race each other.const results: ToolResultBlockParam[] = [];for (const use of toolUses) {  const started = Date.now();  const outcome = await runTool(use.name, use.input, ctx);  results.push({    type: 'tool_result',    tool_use_id: use.id,    content: outcome.content,    ...(outcome.isError ? { is_error: true } : {}),  });  ctx.log.event({    t: 'tool_result', step, tool_use_id: use.id, name: use.name,    is_error: outcome.isError, content: clip(outcome.content), ms: Date.now() - started,  });} // LAB-001.6 One tool_result per tool_use, all in one user message, results first.messages.push({ role: 'user', content: results });

Each line encodes a decision:

  • Append the assistant content unchanged. The documented pattern keeps a running message list and appends the response content before the tool results (build a tool-using agent). If the model returns thinking blocks, pass them back unmodified, particularly during tool use (adaptive thinking). Do not rebuild the turn from the parts you care about.
  • Answer every tool_use. Return one tool_result for each tool_use, all together in the next user message, matched by tool_use_id, with every result before any text (parallel tool use). Claude requested two tools at once in every live run of this lab.
  • Run tools sequentially. Parallel requests do not oblige you to execute in parallel. Sequential execution keeps two writes from racing. If you want parallel reads, parallelize reads only.
  • Name every stop. max_tokens can cut a response off mid tool call; refusal means the model declined. Neither is success (stop reasons). The lab maps anything unexpected, such as pause_turn (used by server tools, which this lab does not have), to unexpected_stop.

F-04, the orphaned tool call. The most common first-draft bug answers only the first tool_use. The API rejects the next request. This is the real error, from a probe we ran directly against the Claude API on 2026-10-17:

API error (probe, 2026-10-17)text
HTTP 400 "messages.2: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_01B.Each `tool_use` block must have a corresponding `tool_result` block in the next message."

The scripted double checks the pairing the same way and fails with the same message, so you find this bug offline. Order matters too: in the same probe, a text block placed before the results was rejected with the same error, naming both ids. The double does not check order, so the LAB-001 tests do. An earlier probe through a gateway accepted text first, which is one more reason to follow the documented rule rather than whatever one endpoint tolerates.

F-03, the runaway loop. With while (true), a model that never stops never stops. The for loop with MAX_STEPS makes the bound structural. You will see in Repair 4 that a cap alone still wastes calls, and what to add.

Every tool_use gets one tool_result. One ticket through the repaired loop (run OFF-06): five model calls, one parallel turn, one approval, then verification. You build the approval in Repair 2 and the verification in Verify the outcome. Silent loop, 14 seconds.
Extended description

Title: "Every tool_use gets one tool_result." Subtitle: "Same id. Every result in one message, before the model is called again."

A five-row table with three columns: "Model asks", "Harness answers", and "What happened". An arrow points from each request to its answer.

1. Model asks A get_ticket; harness answers result A; what happened: read T-1042.

2. Model asks B get_order and C search_policy in the same turn; harness answers result B and result C; what happened: two results, one message.

3. Model asks D issue_refund; harness answers result D; what happened (amber): operator approved, RF-0001 $48.00.

4. Model asks E update_ticket; harness answers result E; what happened: T-1042 resolved.

5. Model stops with end_turn; the harness does not trust it and runs verify the record; what happened (mint): verified_success.

Footer: "Order of calls from run OFF-06 (repaired harness, scripted double). Ids shortened to letters." In the trace the ids are toolu_scripted_001 to toolu_scripted_005.

Read the trace

Every run writes an action log to .runs/: one JSON object per line for each model call, tool result, approval, escalation, and stop. pnpm trace prints it as a timeline. Arrows pointing right are what the model asked for; arrows pointing left are what your harness returned, each with the id it answers.

Observed pnpm trace for the checkpoint A loop, offline, with a lost refund response (run OFF-03, 2026-10-13; long lines shortened):

pnpm trace (run OFF-03)text
step  3  model  stop_reason=tool_use         -> issue_refund({"ticket_id":"T-1042","order_id":"ORD-5521","amount_cents":4800,"reason":"damaged_on_arrival"})  [toolu_scripted_004]         <- issue_refund ERROR  [toolu_scripted_004]  POST /refunds timed out after 3000 msstep  4  model  stop_reason=tool_use         -> issue_refund({"ticket_id":"T-1042","order_id":"ORD-5521","amount_cents":4800,"reason":"damaged_on_arrival"})  [toolu_scripted_005]         <- issue_refund ok  [toolu_scripted_005]  {"kind":"refund","ticket_id":"T-1042","order_id":"ORD-5521","amount_cents":4800, ...step  6  model  stop_reason=end_turn  text="Refunded $48.00 to order ORD-5521 and resolved ticket T-1042."

Read a trace in three passes:

  1. Pairing. Every -> id has exactly one <- with the same id before the next step. The verifier checks this as TRACE-01.
  2. Repetition. Look for the same tool with the same input more than once. Two issue_refund calls with identical inputs, toolu_scripted_004 and toolu_scripted_005, are the signature of a repeated write.
  3. Divergence. Compare what the tools reported with what the model said. Here the first refund call reported a timeout, a second one succeeded, and the final message reports one refund. The trace cannot tell you whether the first call moved money. Only the system of record can.

Break it on purpose

You now have a loop that pairs calls correctly, stops, and passes its tests. Before adding anything, find out what it does when the world misbehaves. The lab injects faults into the simulated services with --inject. Faults change the environment, never the model.

The successful run that fails

Start with no fault. Observed with checkpoint A, offline (run OFF-01, 2026-10-13):

Run OFF-01text
Agent said:   "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042."Harness stop: end_turn after 5 model call(s) $ pnpm verify --ticket T-1042Verification for T-1042 (read from the system of record, not the transcript)  PASS  INV-01  ticket is resolved or escalated: status is "resolved"  PASS  INV-02  payouts are within policy: paid $48.00; POL-DOA-30 allows up to $60.00  PASS  INV-03  at most one payout per ticket: 1 payout(s): RF-0001 $48.00  FAIL  INV-04  every payout has a matching approval: RF-0001 without a matching approved fingerprint  PASS  INV-05  status is consistent with the record: status matches the recorded actions  PASS  TRACE-01  every tool_use was answered exactly once before the next model call: 5 tool_use block(s) requested, all pairedOutcome: failed

The demo worked, and the system is not in an acceptable state: money moved without a human approving it. The system prompt says the system asks an operator to approve each payout. Nothing in the code does. Claude produced the same result in run LIVE-01 on 2026-10-13: right amount, right ticket, no approval. When we repeated that run directly against the Claude API on 2026-10-17, Claude escalated instead, because the order lists only the $249.00 tent and nothing in the record confirmed the $48.00 pole price. Either way, the outcome came from the model's judgment, not from the code.

A malformed tool result (F-01)

--inject policy-truncated makes the policy API cut its response off mid-payload while still answering HTTP 200, the way a misbehaving proxy can. The starter's search_policy passes the body straight to the model. This is exactly what it received:

search_policy response (truncated)text
{"policy_id":"POL-DOA-30","category":"damaged_on_arrival","window_days":30,"max_payout_ce

Observed with the scripted double (run OFF-02):

Run OFF-02text
Agent said:   "Refunded $249.00 to order ORD-5521 and resolved ticket T-1042."  FAIL  INV-02  payouts are within policy: paid $249.00; POL-DOA-30 allows up to $60.00

The double did what an overconfident model does with a missing fact: it guessed generously and refunded the whole tent. The harness let a value that never arrived become a $249.00 decision.

A lost response and a repeated write (F-02)

--inject refund-timeout makes the payments service commit the first refund and then lose the response, so the harness sees a timeout. A timeout tells you that you did not hear back. It does not tell you whether the work happened. Observed with the scripted double (run OFF-03):

Run OFF-03text
Agent said:   "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042."  FAIL  INV-02  payouts are within policy: paid $96.00; POL-DOA-30 allows up to $60.00  FAIL  INV-03  at most one payout per ticket: 2 payout(s): RF-0001 $48.00, RF-0002 $48.00

The message is accurate about the second call and silent about the first. The customer was paid twice.

A dependency that never comes back (F-03)

--inject policy-down makes the policy API answer HTTP 503 every time. The double retries forever. Your MAX_STEPS cap ends the run after 10 model calls, eight of them spent retrying a dead service (run OFF-04). Raise the cap with --max-steps 60 and the double's own safety net trips first: The scripted model was called 50 times and the harness never stopped the loop. (run OFF-05). With a live model, every one of those calls is billed.

Each failure's symptom, evidence, cause in the harness, and repair are in the failure gallery after the lab.

Five cards, F-01 to F-05: malformed tool result, repeated write, runaway loop, orphaned tool call, and confident false completion. Each card gives the symptom, the check that catches it, and the repair.
Five failures, and where each is caught. Every symptom here appeared in a recorded lab run or API probe.
Extended description

Title: "Five failures, and where each is caught." Subtitle: "What each one looks like, the check that catches it, and the repair that prevents it."

Five cards, three in the top row and two centered below. Each card has an icon, an id in amber, a title, and three labeled lines.

F-01, shield icon, "Malformed tool result accepted as fact." Symptom: $249.00 refunded against a $60.00 limit. Caught by: INV-02. Repair: validate results; re-check the limit inside the write tool.

F-02, key icon, "Repeated write after a lost response." Symptom: two $48.00 refunds; the agent reports one. Caught by: INV-03. Repair: idempotency key; read back after a timeout.

F-03, loop icon, "Runaway loop." Symptom: many calls, no progress, rising cost. Caught by: step_cap stop. Repair: a low step cap plus a repeated-failure breaker that hands off.

F-04, wrench icon, "Orphaned tool call." Symptom: HTTP 400 on the next request. Caught by: the 400 names the id. Repair: one tool_result per tool_use, all in one message.

F-05, ledger icon, "Confident false completion." Symptom: a success message over a wrong state. Caught by: pnpm verify. Repair: decide success from the record, not the message.

Footer: "Examples from the lab's recorded runs, 2026-10-13. Check IDs match pnpm verify output." The wording matches the failure gallery table in Break it on purpose.

What the live model did

The double is designed to be wrong. Claude often is not. We ran the same two faults live against the checkpoint A harness, four times each, with Claude Sonnet 5.5 at effort medium on 2026-10-13, then repeated all eight runs directly against the Claude API on 2026-10-17. Both dates produced the same counts.

Live runs against the checkpoint A harness, four per fault on each date (2026-10-13 and 2026-10-17)
FaultWhat Claude didOutcome by the record
Truncated policy (4 runs)Noticed the cut-off payload in every run, retried the policy lookup once, and escalated to a human4 of 4 safe escalations; no payout
Lost refund response (4 runs)Did not retry the refund in any run. In 2 runs it re-read the order, saw refunded_cents: 4800, and resolved correctly. In 2 runs it escalated4 of 4 failed verification, because the refund had no approval (INV-04). The 2 escalations also told the customer the refund was unconfirmed when the record shows it was paid

From run LIVE-02-r4 on 2026-10-13, Claude's own summary: "The refund request timed out, so I don't know whether it went through, and I didn't retry it because that could pay the customer twice." That is careful reasoning. The refund had gone through.

Three conclusions follow, and they shape the rest of the guide:

  1. Good model behavior is not a control. Claude avoided both failures that the double hit. A different model, a different prompt, a different effort setting, or a longer conversation can change that. Four runs per fault on each of two dates is an observation, not a guarantee.
  2. Even careful behavior can leave the record and the message disagreeing. The model could not know whether the refund committed. Neither could the harness, because it had no way to ask. That is a design gap, not a model gap.
  3. Test the harness against a model that makes mistakes. That is the double's job. Then run live to see how the real model uses the controls you built.

Repair 1: validate results and return actionable errors

The decision. A tool result is untrusted input until your code has checked it. When a check fails, tell the model what happened, whether a retry can help, and what to do instead. Anthropic's tool-design guidance recommends specific, actionable error responses over opaque codes or tracebacks (Writing effective tools for agents).

The mechanism. The reference solution validates the policy with a Zod schema (Zod) before anything downstream may rely on it, and turns every failure into a structured error result:

src/tools.ts (validate results)ts
const PolicySchema = z.strictObject({  policy_id: z.string().min(1),  category: z.enum(['damaged_on_arrival', 'late_delivery', 'wrong_item']),  window_days: z.int().positive(),  max_payout_cents: z.int().nonnegative(),  payout_type: z.enum(['refund', 'store_credit', 'replacement']),  summary: z.string(),}); let body: unknown;try {  body = JSON.parse(response.body);} catch {  throw new ToolFailure('invalid_payload', `The policy service returned a response that is not valid JSON (${response.body.length} bytes, likely truncated).`, {    retryable: true,    nextStep: 'Call search_policy again once. If it fails again, call update_ticket with status "needs_human" and an internal note that explains what failed. Never estimate a payout limit.',  });}const parsed = PolicySchema.safeParse(body);if (!parsed.success) throw new ToolFailure('invalid_payload', /* ... */);return parsed.data;

The dispatcher turns a ToolFailure into a tool_result with is_error: true and a JSON body. This is what the model now receives instead of a truncated policy (run OFF-08):

Error result (run OFF-08)json
{"error":"invalid_payload","message":"The policy service returned a response that is not valid JSON (89 bytes, likely truncated).","retryable":true,"next_step":"Call search_policy again once. If it fails again, call update_ticket with status \"needs_human\" and an internal note that explains what failed. Never estimate a payout limit."}

Three choices matter here:

  • retryable is a fact you know and the model does not. A truncated body may succeed on retry; a 503 from a service that is down, or a declined approval, will not. The reference marks HTTP 5xx from the policy service as not retryable.
  • Validation lives in a function the write path also calls. loadPolicy is used by search_policy and by issue_refund. The refund never depends on what the model remembers from an earlier result.
  • strict: true does not do this for you. It guarantees the shape of the model's tool inputs (strict tool use), not the content of your tools' results. Strict schemas also cannot express numeric ranges such as minimum and maximum (structured outputs), so "amount must be positive" is your code's job too.

Repair 2: enforce the write boundary and require approval

The decision. Put every rule that protects money or data in the tool handler, where it runs no matter what the model decides. Then require a human decision before the consequential write, for exactly the action being taken. Anthropic's agent framework makes the same point: humans should retain control before high-stakes decisions, such as an agent canceling subscriptions (framework for safe and trustworthy agents).

The mechanism. Before anyone is asked to approve a refund, checkRefund confirms that the ticket is the one this run was given, that the order and category match the ticket, that the amount is a positive integer, that the policy loaded and validated, that the amount is within the policy limit and the reporting window, and that refunds would not exceed the order total. Then:

src/tools.ts (write boundary)ts
const fingerprint = `refund:${ticketId}:${orderId}:${amount}`; // e.g. refund:T-1042:ORD-5521:4800 // Already declined? Do not ask the operator again for the same action.const prior = ctx.desk.read().approvals.find((a) => a.fingerprint === fingerprint);let decision = prior?.decision;if (!decision) {  const summary = `Refund ${dollars(amount)} on ${orderId} for ${ticketId} (${policy.policy_id}, limit ${dollars(policy.max_payout_cents)})`;  decision = await ctx.approver({ fingerprint, tool: 'issue_refund', summary });  ctx.desk.recordApproval({ fingerprint, tool: 'issue_refund', decision, decided_by: 'operator', summary });}if (decision !== 'approved') {  throw new ToolFailure('approval_denied', 'The operator declined this refund. No money moved.', {    retryable: false,    nextStep: 'Do not request it again. Call update_ticket with status "needs_human" and an internal note that explains what failed.',  });}

What the operator sees (run OFF-06):

Approval prompt (run OFF-06)text
APPROVAL REQUIRED (issue_refund)  Refund $48.00 on ORD-5521 for T-1042 (POL-DOA-30, limit $60.00)  Approve? [y/N]

Design choices worth defending:

  • Check before you ask. An amount over the limit is refused in code and never reaches a human. Approval is for judgment, not for catching what code can catch. The LAB-003 test for this asserts that the operator was asked zero times.
  • Approve an exact action. The fingerprint names the ticket, order, and amount. Approving "a refund" would let a different amount ride on the same decision.
  • Fail closed. The terminal approver treats anything other than y or yes, including an empty line or closed input, as a denial.
  • Not every write needs a human. update_ticket is a write too, but it moves no money, so it has invariant checks (Verify the outcome) instead of an approval step. Classify writes by consequence.
The proposed refund passes code checks and becomes the fingerprint refund:T-1042:ORD-5521:4800, which serves three purposes: the subject of the operator's approval prompt, the idempotency key for the payment, and what the verifier matches for INV-04.
One fingerprint names one action. One string ties the approval, the payment, and the audit together. Repair 3 uses it as the idempotency key and the verifier checks it.
Extended description

Title: "One fingerprint names one action." Subtitle: "The same string is the approval subject, the idempotency key, and what the verifier matches."

Top row, left to right with arrows:

1. "Model proposes": issue_refund, ticket_id: T-1042, order_id: ORD-5521, amount_cents: 4800.

2. "checkRefund, in code" (shield icon): the ticket, order, and reason match this run; the policy loaded and validated; the amount is under the $60.00 limit and the damage was reported within 30 days; otherwise the tool refuses and no human is asked.

3. "Fingerprint" (mint): refund:T-1042:ORD-5521:4800.

A mint line from the fingerprint branches down to three boxes:

1. "1. Approval subject" (amber border), showing the operator prompt from run OFF-06: APPROVAL REQUIRED (issue_refund), Refund $48.00 on ORD-5521 for T-1042 (POL-DOA-30, limit $60.00), Approve? [y/N], and the note "Anything but y or yes is a denial."

2. "2. Idempotency key" (key icon): payments keep one payout per key; after a timeout, read back by key before any retry; a repeat call returns RF-0001.

3. "3. Verifier match" (ledger icon, mint): INV-04, every payout has a matching approval; a payout without an approved fingerprint fails the run.

Footer: "From the repaired harness (labs/solutions/final/src/tools.ts). The operator prompt is recorded in run OFF-06."

Repair 3: make the write idempotent

The decision. If a write can be repeated, by your retry logic or by the model, it needs an idempotency key: an identifier that lets the receiving system recognize a repeat and return the original result instead of acting twice. Payment APIs commonly work this way; Stripe, for example, saves the result of the first request with a key and returns it for retries with the same key (Stripe idempotent requests). An IETF draft for a standard Idempotency-Key header exists but has expired, so treat the header as a convention, not a standard (IETF draft status).

After a timeout, do not guess. Ask the system of record.

src/tools.ts (idempotent write)ts
const existing = ctx.desk.findPayoutByKey(fingerprint);if (existing) return { ...existing, note: 'Already completed earlier in this run. No new refund was made.' };// ... approval, as in SEC-009 ...const request = { ticket_id: ticketId, order_id: orderId, amount_cents: amount, reason, idempotency_key: fingerprint };try {  return await ctx.services.createRefund(request);} catch (error) {  if (!(error instanceof TimeoutError)) throw error;  // The response was lost, not necessarily the write. Ask the system of record.  const found = await ctx.services.findPayout(fingerprint);  if (found) return { ...found, note: 'The first response timed out; this refund was confirmed by read-back.' };  // Not recorded, so a retry with the SAME key is safe: the server deduplicates on it.  const retried = await ctx.services.createRefund(request);  return { ...retried, note: 'The first attempt timed out before it was recorded; retried once with the same idempotency key.' };}

The fingerprint from Repair 2 doubles as the key, which is what lets the verifier match each payout to its approval later.

A timeout means unknown, not failed. Runs OFF-03 and OFF-09. After a timeout, ask the system of record before acting again. Silent loop, 12 seconds.
Extended description

Title: "A timeout means unknown, not failed." Subtitle: "The same lost refund response, without and with an idempotency key."

Two rows of boxes joined by arrows, read left to right.

Row 1, amber label "Without a key (checkpoint A)":

1. "Refund sent": RF-0001 committed.

2. "Response lost" (amber): the model repeats the call.

3. "Second refund" (amber): RF-0002 committed.

4. Result box (amber): "$96.00 paid", INV-03 FAIL.

Row 2, mint label "With a key and read-back (repaired)":

1. "Approved": RF-0001 sent with its key.

2. "Response lost" (amber): the harness reads back by key.

3. "Found RF-0001" (mint): no second write.

4. Result box (mint): "$48.00 paid once", INV-03 PASS.

Footer: "Runs OFF-03 and OFF-09 (scripted double, 2026-10-13). Simulated payments timeout: 3000 ms." In both runs the first refund was committed before its response was lost; only the second run asked the system of record before acting again.

Choosing the key. This lab derives the key from the action's content, because the business rule is one payout per ticket anyway. Where two identical actions can both be legitimate, generate a key when the action is proposed, store it with the proposal, and reuse it for every retry of that proposal. Either way, the key belongs to the action, not to the HTTP attempt.

Repair 4: stop on repetition and hand off

The decision. A step cap bounds the damage; it does not recognize a loop that has stopped making progress. Add a breaker: when the same tool fails twice with the same input, stop. Then give the work to a person, through code the harness owns. Anthropic's guidance describes agents pausing for human feedback at checkpoints or when they hit blockers (Building effective agents).

src/agent.ts (repetition breaker)ts
if (outcome.isError) {  const key = `${use.name}:${stableJson(use.input)}`;  const count = (failures.get(key) ?? 0) + 1;  failures.set(key, count);  if (count >= MAX_IDENTICAL_FAILURES) tripped = `${use.name} failed ${count} times with the same input`;}// ... after sending the results:if (tripped) {  escalate(ctx, 'repeated_failure', tripped);  return { stop: 'repeated_failure', steps: step, finalText: '', messages };}

escalate sets the ticket to needs_human with an internal note that names the reason. The same call runs on step_cap, max_tokens, refusal, any unexpected stop, and an end_turn that leaves the ticket open. A run never ends with the ticket unowned.

With the dead policy service, the repaired harness stops after 3 model calls instead of 10 (run OFF-10). Live, with the truncated policy, Claude retried once as the error told it to, and the breaker ended the run at the second failure (run LIVE-06). The cooperative path and the enforced path stop in the same place, which is the point: the enforced one does not depend on cooperation.

Choosing the threshold. Two identical failures suits a short task with a cheap, honest error signal. Raise it for flaky reads you expect to recover. Keep identical-input matching: different inputs mean the model is changing its approach, which a breaker should not punish.

Verify the outcome, not the transcript

The decision. Success is a property of the system of record, checked by code that never saw the conversation. The model's last message is a claim about the outcome, written by the component least able to know it. Anthropic's guidance on agent evaluation draws the same line: the transcript is what the agent said and did, and the outcome is the final state of the environment (Demystifying evals for AI agents).

The mechanism. src/verify.ts reads .state/desk.json from disk and checks invariants the business cares about:

The verifier invariants (pnpm verify)
CheckInvariantCatches
INV-01The ticket ended resolved or needs_humanA run that stopped with no owner
INV-02Payouts are within the policy limit and of the policy's payout typeF-01
INV-03At most one payout per ticketF-02
INV-04Every payout has a matching approved fingerprintUnapproved writes
INV-05The status agrees with the record: a resolved refund ticket has exactly one payout, an escalation has an explanatory noteF-05
TRACE-01Every tool_use was answered exactly once before the next model callF-04

The repaired CLI runs these checks itself after every run and reports the verified outcome next to the agent's message. Observed with the scripted double and a lost refund response (run OFF-09):

Run OFF-09text
Agent said:   "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042."Harness stop: end_turn after 5 model call(s) Verified against the system of record:  PASS  INV-01  ticket is resolved or escalated: status is "resolved"  PASS  INV-02  payouts are within policy: paid $48.00; POL-DOA-30 allows up to $60.00  PASS  INV-03  at most one payout per ticket: 1 payout(s): RF-0001 $48.00  PASS  INV-04  every payout has a matching approval: 1 payout(s), all approved  PASS  INV-05  status is consistent with the record: status matches the recorded actions  PASS  TRACE-01  every tool_use was answered exactly once before the next model call: 5 tool_use block(s) requested, all pairedOutcome:      verified_success

The agent's message is word for word the one it produced when it paid twice. The message did not change. The outcome did.

There are three outcomes, and the difference matters when you report results:

  • verified_success: every invariant holds and the ticket is resolved.
  • safe_escalation: every invariant holds and a human owns the ticket. For a fault you cannot fix in code, this is the correct result, not a failure.
  • failed: an invariant is violated. Something happened that should not have.

The repaired harness, across the whole fault suite offline and live (2026-10-13; the live runs were repeated directly against the Claude API on 2026-10-17 with the same stops and outcomes):

The repaired harness across the fault suite
RunFaultStopOutcome
OFF-06, LIVE-04Noneend_turnverified_success
OFF-07Policy truncated onceend_turnverified_success
OFF-08, LIVE-06Policy always truncatedrepeated_failuresafe_escalation
OFF-09, LIVE-05Lost refund responseend_turnverified_success, one refund
OFF-10, LIVE-07Policy service downrepeated_failure offline, end_turn livesafe_escalation
OFF-11, LIVE-08Operator declinesrepeated_failure offline, end_turn livesafe_escalation, no refund
Two panels with the identical agent message, "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042." Before the repair the record holds two $48.00 payouts and the outcome is failed. After the repair it holds one approved payout and the outcome is verified_success.
Same sentence, different record. Runs OFF-03 and OFF-09 ended with the same sentence. Only the record tells them apart.
Extended description

Title: "Same sentence. Different record." Subtitle: "The agent's final message did not change. The outcome did."

Two side-by-side panels.

Left panel, amber border, "Before the repair", run OFF-03, refund-timeout.

Agent said: "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042."

The record: RF-0001 $48.00, RF-0002 $48.00.

INV-03 FAIL 2 payouts; outcome: failed.

Right panel, mint border, "After the repair", run OFF-09, refund-timeout.

Agent said: the identical sentence.

The record: RF-0001 $48.00 approved.

INV-03 PASS 1 payout; outcome: verified_success.

Footer: "Rendered from recorded lab runs OFF-03 and OFF-09 (offline scripted double, 2026-10-13)." The sentence is copied from agent-output.txt in both run folders.

Transfer it: the store-credit assessment

Repairing the refund path proves you can follow this guide. The assessment checks whether you can apply it to something new. Ticket T-2077 is a late delivery, and the policy POL-LATE-14 pays out as store credit, up to $15.00, only for orders that arrived after their promised date. You add an issue_store_credit tool with its own contract, the same protections as the refund, an eligibility rule enforced at the write boundary, and a new verifier invariant, INV-06. Your instructor scores it with tests you have not seen and a rubric you have. The task statement and rubric are in the assessment brief your instructor provides.

What this lab simplifies

The lab teaches the decisions with the smallest system that shows them. Before you apply the same design in production, account for what it leaves out:

  • The system of record is a local file. Real writes are network calls to systems with their own consistency rules, and read-back after a timeout may itself be eventually consistent. Check the target system's documented guarantees.
  • The idempotency store and the payment are the same file. In production, the payment provider must honor the key. If it does not, you need your own outbox or ledger in front of it.
  • Approval is a terminal prompt. Production approval needs authenticated approvers, an audit trail, timeouts, and a policy for what happens when nobody answers.
  • One ticket, one run, one process. Concurrent runs on the same ticket need locking or a queue.
  • Synthetic, cooperative inputs. The ticket text is harmless. Untrusted text that tries to steer the agent is a separate problem; the next guides on containment and prompt injection cover it.
  • One model, one effort level, small samples. The live observations are four runs per fault on each of two dates. Measuring an agent properly is the subject of a separate evaluation guide.

Troubleshooting

Troubleshooting
SymptomLikely causeFix
LAB-001: the loop body is not implemented yetYou are running the untouched starterImplement the TODOs in src/agent.ts
400 messages.N: tool_use ids were found without tool_result blocks...A tool_use went unanswered, usually the second of two in one turnLoop over every tool_use block and send all results in one message (Build the bounded loop)
400 ... unexpected tool_use_id found in tool_result blocksA result answers an id from an older turn, or a typo in tool_use_idBuild results from the current response's tool_use blocks only
No desk state yet. Run "pnpm reset" first.Fresh clone or deleted .state/pnpm reset
ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX or Unknown file extension ".ts"Node.js older than 22.18, or an enum or parameter property in your codenode --version; upgrade; replace enums with string unions
Cannot find module on an importMissing .ts extension in a relative importNode requires the extension: import './tools.ts'
Live mode needs ANTHROPIC_API_KEYNo key in this shellExport the key, or keep using --model scripted
HTTP 401 or 403 liveWrong, revoked, or unauthorized keyCheck the key and workspace in the Claude Console; do not retry with another credential
HTTP 429 or 529 liveRate limit or overloadThe SDK retries automatically; wait and run again (errors)
pnpm install fails on a version that is "too new"pnpm's minimumReleaseAge policyUse the committed lockfile with pnpm install --frozen-lockfile; do not add exclusions to get around it
The approval prompt does not wait for inputInput is piped or closedExpected: a closed input is a denial. Type y at the prompt, or pipe y for scripted runs
Run crashed: The scripted model was called 50 times...The loop never stoppedYour step cap is missing, or set above 50
A live run passes and the offline test failsClaude handled a fault the double does notThe harness must pass offline; the live run shows model behavior, not harness guarantees (What the live model did)

Quick reference

The loop:

  • Append response.content unchanged, including thinking blocks.
  • end_turn ends the loop. It does not prove success.
  • Any stop reason other than tool_use is a stop, never success.
  • One tool_result per tool_use, matched by tool_use_id, all in one user message, results first.
  • Run tools sequentially unless you have made concurrent writes safe.
  • Bound the loop structurally (MAX_STEPS), and stop on repeated identical failures.

The tool boundary:

  • Validate every tool result before the model relies on it. strict: true covers inputs, not results, and not ranges.
  • Return failures as is_error: true with what happened, whether a retry is safe, and the next step.
  • Re-check business rules inside the write handler. Do not trust amounts or limits restated by the model.
  • Consequential writes need an approval for the exact action, checked after the code checks pass.
  • Every repeatable write carries an idempotency key; after a timeout, read back before retrying.

The outcome:

  • Decide success from the system of record, read by code that never saw the conversation.
  • Report verified_success, safe_escalation, or failed, and never the model's message alone.

Commands:

Quick reference commandsbash
pnpm reset                                           # restore the synthetic datapnpm agent --ticket T-1042                           # offline, scripted doublepnpm agent --ticket T-1042 --model claude            # live (needs ANTHROPIC_API_KEY)pnpm agent --ticket T-1042 --inject refund-timeout   # seeded faultpnpm trace                                           # timeline of the last runpnpm verify --ticket T-1042                          # independent verificationpnpm test:lab1 && pnpm test:lab3                     # acceptance tests

Lab 1: Build the bounded loop

Write the loop body in src/agent.ts (TODOs LAB-001.1 to LAB-001.6): append each assistant turn exactly as received, return on end_turn and a named stop for every other stop reason, and answer every tool_use with one tool_result, all in one user message. About 20 minutes, offline against the scripted double. Outputs show the lines to compare: your terminal also prints a run id and trace path, and the test runner adds timings and stack lines.

Acceptance checks

  • pnpm test:lab1 reports 5 passing and 0 failing
  • pnpm agent --ticket T-1042 ends with end_turn after 5 model calls
  • Every tool_use id in pnpm trace is answered exactly once before the next step
  1. Run the untouched starter

    $ pnpm agent --ticket T-1042

    Initial: starter code

    Run crashed: LAB-001: the loop body is not implemented yet. Open src/agent.ts.

    Final: with your Lab 1 loop

    Agent said: "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042." Harness stop: end_turn after 5 model call(s) The message above is a claim. Check the record: pnpm verify --ticket T-1042
  2. Implement the loop, then run its tests

    $ pnpm test:lab1

    Initial: starter code

    not ok 1 - LAB-001 happy path: the loop runs tools until the model ends its turn error: 'LAB-001: the loop body is not implemented yet. Open src/agent.ts.' not ok 2 - LAB-001 two tool calls in one turn get two results in one message (F-04) error: 'LAB-001: the loop body is not implemented yet. Open src/agent.ts.' not ok 3 - LAB-001 the assistant turn is appended exactly as the model returned it error: 'LAB-001: the loop body is not implemented yet. Open src/agent.ts.' not ok 4 - LAB-001 a model that never stops is stopped by the step cap (F-03) error: 'LAB-001: the loop body is not implemented yet. Open src/agent.ts.' not ok 5 - LAB-001 max_tokens and refusal end the loop instead of being treated as success error: 'LAB-001: the loop body is not implemented yet. Open src/agent.ts.' # tests 5 # pass 0 # fail 5

    Final: with your Lab 1 loop

    ok 1 - LAB-001 happy path: the loop runs tools until the model ends its turn ok 2 - LAB-001 two tool calls in one turn get two results in one message (F-04) ok 3 - LAB-001 the assistant turn is appended exactly as the model returned it ok 4 - LAB-001 a model that never stops is stopped by the step cap (F-03) ok 5 - LAB-001 max_tokens and refusal end the loop instead of being treated as success # tests 5 # pass 5 # fail 0
  3. Run the agent on the ticket

    $ pnpm reset && pnpm agent --ticket T-1042

    Initial: with your Lab 1 loop

    Agent said: "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042." Harness stop: end_turn after 5 model call(s) The message above is a claim. Check the record: pnpm verify --ticket T-1042

    Final: after the Lab 3 repairs, approved with y

    APPROVAL REQUIRED (issue_refund) Refund $48.00 on ORD-5521 for T-1042 (POL-DOA-30, limit $60.00) Approve? [y/N] y Agent said: "Refunded $48.00 to order ORD-5521 and resolved ticket T-1042." Harness stop: end_turn after 5 model call(s) Verified against the system of record: PASS INV-01 ticket is resolved or escalated: status is "resolved" PASS INV-02 payouts are within policy: paid $48.00; POL-DOA-30 allows up to $60.00 PASS INV-03 at most one payout per ticket: 1 payout(s): RF-0001 $48.00 PASS INV-04 every payout has a matching approval: 1 payout(s), all approved PASS INV-05 status is consistent with the record: status matches the recorded actions PASS TRACE-01 every tool_use was answered exactly once before the next model call: 5 tool_use block(s) requested, all paired Outcome: verified_success
  4. Check the record, not the message

    $ pnpm verify --ticket T-1042

    Initial: with your Lab 1 loop

    Verification for T-1042 (read from the system of record, not the transcript) PASS INV-01 ticket is resolved or escalated: status is "resolved" PASS INV-02 payouts are within policy: paid $48.00; POL-DOA-30 allows up to $60.00 PASS INV-03 at most one payout per ticket: 1 payout(s): RF-0001 $48.00 FAIL INV-04 every payout has a matching approval: RF-0001 without a matching approved fingerprint PASS INV-05 status is consistent with the record: status matches the recorded actions PASS TRACE-01 every tool_use was answered exactly once before the next model call: 5 tool_use block(s) requested, all paired Outcome: failed

    Final: after the Lab 3 repairs

    Verification for T-1042 (read from the system of record, not the transcript) PASS INV-01 ticket is resolved or escalated: status is "resolved" PASS INV-02 payouts are within policy: paid $48.00; POL-DOA-30 allows up to $60.00 PASS INV-03 at most one payout per ticket: 1 payout(s): RF-0001 $48.00 PASS INV-04 every payout has a matching approval: 1 payout(s), all approved PASS INV-05 status is consistent with the record: status matches the recorded actions PASS TRACE-01 every tool_use was answered exactly once before the next model call: 5 tool_use block(s) requested, all paired Outcome: verified_success
  5. Show the solution discussion

    The reference loop is labs/solutions/checkpoint-a/src/agent.ts in the lab download, and the lesson shows the part you write in Build the bounded loop.

    Two details matter most. Push the assistant content back unchanged, because thinking blocks and tool_use ids must survive the round trip. Then collect every result for the turn before calling the model again, so a parallel turn gets one user message with one tool_result per id.

    The failing verify in the last step is expected at this point: the loop works, but nothing yet asks a person to approve the refund. Lab 3 adds that control, and the Final output of steps 3 and 4 shows the same commands once it is in place.

More labs in this guide

  • Lab 2: Break it and diagnose

    Run the three seeded faults (policy-truncated, refund-timeout, policy-down) against your own loop, fill in a diagnosis row for each run with the agent's quote, the verified outcome, and trace evidence, and name the harness behavior that caused it. About 15 minutes.

  • Lab 3: Repair and verify

    Add the four repairs from the lesson (validated results, a write boundary with approval, an idempotent refund with read-back, and a repeated-failure breaker), then run the whole fault suite until every run ends in verified success or a safe escalation. About 30 minutes, with hints.

Five ways this breaks, and how you will know

Every failure here is reproducible in the lab with a flag. Learn the symptom first; it is what you will actually see.

  • F-01--inject policy-truncatedReproducible

    Malformed tool result accepted as fact

    Symptom
    A payout far above the policy limit: $249.00 against a $60.00 limit.
    Diagnosis
    Results reach the model unvalidated, and the write path trusts the model's amount. The trace shows a cut-off search_policy body marked ok.
    Fix
    Validate every result before the model sees it, and re-check the limit inside the write tool (Repairs 1 and 2).
    Verify
    INV-02 passes; the truncated policy now ends in a safe escalation with no payout.
  • F-02--inject refund-timeoutReproducible

    Repeated write after a lost response

    Symptom
    Two identical $48.00 payouts, while the agent reports one.
    Diagnosis
    No idempotency key, and a timeout is treated as "it failed" instead of "unknown."
    Fix
    Use the action fingerprint as an idempotency key, and read back by key after a timeout (Repair 3).
    Verify
    INV-03 passes with exactly one payout.
  • F-03--inject policy-downReproducible

    Runaway loop

    Symptom
    Many calls, no progress, rising cost.
    Diagnosis
    No stop on repetition; a cap that is too high is no cap. Eight of ten calls retried a dead service.
    Fix
    A low step cap plus a repeated-failure breaker that hands off to a person (Repair 4).
    Verify
    The run stops with repeated_failure after two identical errors, and the ticket is set to needs_human.
  • F-04Reproducible

    Orphaned tool call

    Symptom
    HTTP 400 on the next request.
    Diagnosis
    Only some tool_use blocks got a tool_result; the 400 message names the unanswered id.
    Fix
    Answer every tool_use with one tool_result, all in one user message, before any other content.
    Verify
    pnpm test:lab1 passes, including the two-tools-in-one-turn test.
  • F-05Reproducible

    Confident false completion

    Symptom
    A success message over a wrong state.
    Diagnosis
    Success is read from the transcript, not from the outcome in the system of record.
    Fix
    Decide success with an independent verifier that reads the record (Verify the outcome).
    Verify
    pnpm verify reports verified_success, or the run is escalated to a person.

Transfer task and rubric

Extend your Lab 3 agent with a store-credit tool that carries every refund protection, then prove each run from the record.

Analytic rubric: each criterion described at four performance levels, from Emerging to Exemplary
Criterion0 · Emerging1 · Developing2 · Proficient3 · Exemplary
C1 Tool contractNo new tool, or a schema that accepts any inputTool exists but is not strict, not closed, or accepts reasons other than late_deliveryStrict, closed, integer amount, reason limited to late_delivery, description states the consequenceProficient, and the description tells the model when not to use the tool (for example, not for damaged items)
C2 Correct resolutionNo credit, or the run fails verificationA credit is issued but the amount, approval, or ticket status is wrongOne approved $15.00 credit, ticket resolved, verified_successProficient, and the customer reply and internal note state the policy and amount accurately
C3 Enforced protections (critical)A credit can be issued without approval, above the limit, or twiceSome protections exist but at least one is missing or can be bypassedLimit and payout type checked in code before approval; approval for the exact action; idempotency key; read-back after timeout; refunds unchangedProficient, implemented by generalizing the refund path rather than duplicating it, with no behavior change to refunds
C4 Eligibility and invariantNeither the write path nor the verifier checks latenessOne of the two checks existsWrite path refuses on-time orders before approval, and INV-06 fails a credit on an on-time orderProficient, and INV-06's detail line shows the promised and delivered dates
C5 EvidenceOutputs missingSome outputs missing or inconsistent with the codeAll required outputs present and consistent with the submitted codeProficient, with a one-line note per run explaining the outcome

Proficiency standard: Proficient or higher on every criterion. Proposed local standard, not an official certification.

Self-check

The full brief

The scenario: ticket T-2077 is a late delivery. Order ORD-6610, a $69.00 camp stove, was promised for October 3 and delivered October 7, and the customer asks for the $15.00 late-delivery credit. Policy POL-LATE-14 pays store credit up to $15.00 when an order arrives after its promised date and is reported within 14 days of delivery. It never pays cash for lateness.

What you build, starting from your Lab 3 code: an issue_store_credit tool with a strict, closed schema (ticket_id, order_id, an integer amount_cents, and a reason limited to late_delivery) and a description that says it is a consequential write. It gets every protection the refund has: the policy loaded and validated in code, the payout type and limit enforced, the reporting window and the ticket's own order and category checked, approval for one exact action, that action's fingerprint as the idempotency key, and a read-back after a timeout. The tool refuses the credit, in code and before approval, unless the order was late, and the verifier gains INV-06: a store credit exists only for a late order.

Constraints and evidence: all 12 lab tests must still pass, and the refund tool must not be loosened. Submit your changed files, the pnpm test and pnpm typecheck output, and the agent and pnpm verify output for three runs: the plain T-2077 run and the credit-timeout run, both approved with y, and the policy-truncated run. Then answer two short written items: read two traces, and explain an outcome from your own verify output.

Timing and scoring: about 30 minutes to build, then about 10 minutes for the written items. Your instructor runs tests you have not seen and scores the rubric above. The proficiency standard, a proposed local standard, is Proficient or higher on every criterion with no critical failure, meaning no run in which money moved without a matching approval or above the policy limit.

Teach it to a room

  • Facilitation guide and run of show

    A 90-minute workshop plan with timings, the misconceptions to listen for, and what to do when a live demo fails.

  • Cohort preparation and checkpoints

    A fallback environment for machines that cannot install packages, and checkpoint code for the loop and the final repairs, so a stalled learner can rejoin at the next lab.

  • Dry run and observation rubric

    A train-the-trainer sequence, a 60-minute dry-run protocol, and an observation rubric for an instructor teaching the module for the first time.

  • Module clearance record

    A proposed local standard and a blank record for deciding that an instructor is cleared to teach this module. It is a new template; no instructor has been cleared with it.

  • Evaluation plan and pass-rate report

    Diagnostic, exit, and feedback questions, data handling rules, and a blank monthly pass-rate report. No learner or instructor results exist yet.

  1. [1]Tool use overview · platform.claude.comchecked Oct 13, 2026
  2. [2]How tool use works · platform.claude.comchecked Oct 13, 2026
  3. [3]Build a tool-using agent · platform.claude.comchecked Oct 13, 2026
  4. [4]Handle tool calls · platform.claude.comchecked Oct 13, 2026
  5. [5]Parallel tool use · platform.claude.comchecked Oct 13, 2026
  6. [6]Handling stop reasons · platform.claude.comchecked Oct 13, 2026
  7. [7]Define tools · platform.claude.comchecked Oct 13, 2026
  8. [8]Strict tool use · platform.claude.comchecked Oct 13, 2026
  9. [9]Structured outputs · platform.claude.comchecked Oct 13, 2026
  10. [10]Models overview · platform.claude.comchecked Oct 13, 2026
  11. [11]Effort · platform.claude.comchecked Oct 13, 2026
  12. [12]Adaptive thinking · platform.claude.comchecked Oct 13, 2026
  13. [13]Pricing · platform.claude.comchecked Oct 13, 2026
  14. [14]TypeScript SDK · platform.claude.comchecked Oct 13, 2026
  15. [15]Errors · platform.claude.comchecked Oct 13, 2026
  16. [16]TypeScript SDK repository · github.comchecked Oct 13, 2026
  17. [17]@anthropic-ai/sdk · npmjs.comchecked Oct 13, 2026
  18. [18]Building effective agents · anthropic.comchecked Oct 13, 2026
  19. [19]Demystifying evals for AI agents · anthropic.comchecked Oct 13, 2026
  20. [20]Writing effective tools for agents · anthropic.comchecked Oct 13, 2026
  21. [21]Our framework for developing safe and trustworthy agents · anthropic.comchecked Oct 13, 2026
  22. [22]Modules: TypeScript (v22) · nodejs.orgchecked Oct 13, 2026
  23. [23]Release schedule · github.comchecked Oct 13, 2026
  24. [24]TSConfig reference: erasableSyntaxOnly · typescriptlang.orgchecked Oct 13, 2026
  25. [25]Mitigating supply chain attacks · pnpm.iochecked Oct 13, 2026
  26. [26]Defining schemas · zod.devchecked Oct 13, 2026
  27. [27]Idempotent requests · docs.stripe.comchecked Oct 13, 2026
  28. [28]The Idempotency-Key HTTP Header Field (expired draft) · datatracker.ietf.orgchecked Oct 13, 2026

Independent work. Not affiliated with or endorsed by any vendor named in this guide.

Jordan Hudgens

Written by

Jordan Hudgens

Technical educator and AI engineering leader. 16 Udemy courses, 111K students, and curricula used by 250K+ learners.

Contact Jordan

Next in the path

All guides