Skip to content
Jordan Hudgens
Illustration: a small white capsule robot at a dark navy workbench holds a triangular plug. On its left lies a tangle of mismatched, amber-glowing adapters; on its right, a machined plate has three clean sockets: round, square, and triangular.AI-generated illustration
Guide 11IntermediateAgents and tool useEvaluation and testingPrompting and structured output

Design Tool Contracts Agents Can Use Correctly

Turn a confusing tool library into contracts a model reads correctly, then prove the difference on the same tasks with the system of record, not the reply.

Reading
69 min
Workshop
125 min
Stack
TS · Claude API 2026-10-21 · @anthropic-ai/sdk 1.8.0 · Node.js 22 · pnpm 12
Version
1.1.0
Last verified
Nov 16, 2026

What you will build and be able to do

Each outcome has an observable criterion. The lab and the assessment check exactly these, nothing more and nothing less.

  1. OBJ-001

    Diagnose a tool-contract defect from a transcript

    Diagnose a tool-contract defect from a transcript: name the defect class, cite the step and tool call, and name the contract change that removes it.

  2. OBJ-002

    Assign one owning tool to each capability

    Assign one owning tool to each capability, with namespaced names and descriptions that say when to use the tool and when not to.

  3. OBJ-003

    Write input schemas a model cannot misread

    Write input schemas a model cannot misread: specific names, integer cents, enums for closed sets, id patterns, and strict mode.

  4. OBJ-004

    Return decision-sized results and actionable errors

    Return results sized for the next decision and errors that state the input at fault, a valid value, whether a retry can help, and the next step.

  5. OBJ-005

    Put side effects in the contract and enforce them in code

    Put side effects in the contract: reads that change nothing, writes that are approved, idempotent, and checked against policy in code.

  6. OBJ-006

    Compare libraries on outcomes and cost, and decide on discovery

    Compare two tool libraries on the same tasks with outcome checks and cost metrics, and decide from your own numbers whether tool discovery or programmatic calling is justified.

Unpack the download and check your environment ($0.00)

  • A working tool-use loop. You have built or read a loop where the model returns tool_use blocks and your code returns tool_result blocks. If not, start with Guide 01; this guide reuses its synthetic support desk and its bounded harness.Start with Guide 01
  • TypeScript and JSON Schema. If you write Python, Go, or Rust, every contract rule translates directly.Read the contract primer
  • Node.js 22.18.0 or later and pnpm 12. The download pins Node.js 22.23.5 (.node-version, Maintenance LTS with end of life on 2027-04-30, Node.js release index) and pnpm 12.21.0 (packageManager); the offline suite was also verified on Node.js 24.22.3 and Active LTS 26.13.1 on 2026-11-15.pnpm installation guide
  • Optional Anthropic API key for live runs. Every lab and test runs offline at $0.00 with a scripted model double. Live comparisons in LAB-003 use @anthropic-ai/sdk 1.8.0 (Apache-2.0, default API version 2026-10-21) and cost well under a dollar per one-repetition comparison at 2026-11-15 list prices.

Anatomy of a tool contract: three model-read parts, three code-enforced parts

The model reads the name, description, and input schema before it calls; your handler code owns the result shape, error shape, and side effects.
Diagram: two regions. What the model reads holds three boxes: Name (resource first, then verb; one name per responsibility), Description (what it does, when to use it, when not to; units, limits, and what it changes), and Input schema (strict true, additionalProperties false, amount_cents integer, enums, id patterns). What your code returns and enforces holds Result (only what the next decision needs; concise by default), Error (code, field, expected, retryable; next_step), and Effect (declared read or write; approved, idempotent, checked in code).
Extended description

Left region, What the model reads, holds three stacked contract parts sent in the request: 1. Name (payouts_issue, domain first then verb, one name per capability); 2. Description (what it does, when to use it, when not to, units, policy caps, and state changes); and 3. Input schema (strict true, additionalProperties false, integer amount_cents, closed enums, and regex identifier patterns).

Right region, What your code enforces, holds three stacked runtime parts owned by handler code: 4. Result (only what the next decision needs, concise by default and detailed on request, with zero customer contact fields); 5. Error (stable code, field, expected valid value, honest retryable boolean, and next_step naming the recovery tool); and 6. Effect (declared read or write, pure reads, and approved, idempotent, policy-checked writes).

The contract is the interface

A tool contract is everything that defines how a tool can be used: its name, its description, its input schema, the shape of its results, the shape of its errors, and its effect on the world (whether it only reads, or changes something). The model sees the first three in the request and the next two as tool_result content. It never sees your handler code. Whatever the contract leaves unstated, the model has to guess.

That makes tool design an interface-design problem with an unusual consumer. A human developer reads your API reference once, asks a colleague, and remembers. A model reads the contract on every request, with no memory of the last integration and no one to ask. Anthropic's engineering guidance states the failure mode directly: "More tools don't always lead to better outcomes. A common error we've observed is tools that merely wrap existing software functionality or API endpoints" (Writing effective tools for AI agents).

The engineering decision this guide teaches is provider-independent: design each tool around one decision the agent has to make, and make every fact that decision needs explicit in the contract. Here the mechanism is Claude's direct tool-use API, and the same six contract parts appear when a server publishes tools over the Model Context Protocol (Guide 04).

Two rules keep the rest of the guide honest:

  1. The record decides success. A run passes only when the desk's own data shows the right payouts and the right ticket status. The model's final message is a claim. That is the rule Guide 01 builds into the harness, and it is why the baseline runs below recorded zero wrong payouts: the harness caught every bad finish.
  2. Measure on your tasks. Every number in this guide comes from six synthetic tickets, one model (claude-sonnet-5-5), and one effort setting (medium). They show the size of the effects on this fixture, not a universal benchmark.

The lab: one desk, two libraries

The lab extends the synthetic support desk from Guide 01: Example Outfitters, a fictional outdoor retailer with tickets, orders, policies, synthetic @example.com customer profiles, shipping events, inventory, and a write journal. The journal records which tool made every change, so a test can prove that a read changed nothing.

The harness is Guide 01's repaired loop. It caps model calls at 10, stops after two identical failed calls, answers every tool_use with one tool_result placed first in the user content array ("the tool_result blocks must come FIRST in the content array. Any text must come AFTER all tool results.", Handle tool calls), and hands the ticket to a human on any stop that is not a verified finish. Only the tool library changes between runs.

The six Example Outfitters support tickets and their expected outcomes on the system of record.
TicketCustomer asks forPolicyCorrect outcome
T-1042$48.00 for a snapped tent poleDamaged on arrival, $60.00 limit, 30 daysRefund 4800 cents, resolved
T-2077$15.00 credit for a late stoveLate delivery, store credit up to $15.00Store credit 1500 cents, resolved
T-3010The right daypackWrong item: replacement through a personNo payout, needs_human
T-3022$75.00 for a cracked cooler lidOver the $60.00 limitNo payout, needs_human
T-3031$30.00, reported 46 days after deliveryOutside the 30-day windowNo payout, needs_human
T-3045$12.50 for a headlamp strapDamaged on arrivalRefund 1250 cents, resolved

Unpack the download (tool-contracts-labs.zip) and check the starter offline ($0.00):

Offline preflightbash
cd labs/starterpnpm install --frozen-lockfilepnpm preflight

pnpm preflight checks your Node.js version and ends with Ready. plus an INFO line when no API key is set. To run the optional live commands in LAB-003, export your own key in your shell and hold effort constant:

Live env (optional)bash
export ANTHROPIC_API_KEY="<your-claude-console-key>"   # never commit a keyexport AGENT_EFFORT=medium                             # explicit choice; Sonnet 5.5 defaults to high

In API version 2026-10-21 (@anthropic-ai/sdk 1.0.0 and later), sampling parameters temperature, top_p, and top_k return HTTP 400 (temperature: Extra inputs are not permitted, Migrating to API version 2026-10-21). Instead, the lab sets output_config: { effort: "medium" } as an engineering choice: Anthropic's effort guide notes high is the API default on Sonnet 5.5 and advises, "For agentic coding and multistep tool use, start with medium for well-specified tasks and move to high for harder or longer ones." (Effort). Hold effort constant whenever you compare two libraries.

Offline, every command uses a scripted model double (Offline: scripted model double, not Claude): a deterministic stand-in whose five rules sit at the top of src/model/scripted.ts. It copies an amount as the customer wrote it unless the parameter name or description says cents, and when an error gives no valid value it tries the next overlapping tool. Passing with the double proves your contracts removed those structural triggers; only live runs measure model behavior.

Diagnose the baseline

Read src/contracts/baseline.ts before running anything. It holds nine thin endpoint wrappers: search, get_ticket, get_order, get_policy, issue_refund, create_refund_v2, add_credit, update_ticket, and close_ticket. For example, get_policy asks for name while matching only internal IDs such as POL-DOA-30:

baseline.tsts
const getPolicy: ToolContract = {  effect: 'read',  capabilities: ['policy.read'],  definition: {    name: 'get_policy',    description: 'Returns a policy.',    input_schema: { type: 'object', properties: { name: { type: 'string' } }, required: ['name'] },  },  async handler(input, ctx) {    const policy = ctx.desk.read().policies.find((p) => p.policy_id === str(input.name));    if (!policy) return { content: '{"error":"NOT_FOUND"}', isError: true };    return { content: JSON.stringify(policy), isError: false };  },};

Run the static contract linter and the offline evaluation ($0.00):

Baseline checksbash
pnpm lint:contracts --library baselinepnpm eval --library baseline

The linter fails all eight static rules (C-01 to C-08). Offline, the scripted double passes 2 of 6 tickets (4 wrong outcomes): it refunds 48 cents instead of $48.00 on T-1042 (F-03), and when update_ticket rejects escalated it falls back to close_ticket and marks three escalation tickets resolved (F-04). Claude failed differently, and its transcripts are more instructive.

What Claude did with the baseline

We ran claude-sonnet-5-5 at effort medium on the baseline library, 6 tickets times 3 repetitions (18 runs), through the direct Claude API on 2026-11-15 (Synthetic data. Simulated services.). Six of 18 runs passed. Twelve ended in a safe stop, where the harness set needs_human after checking the desk record. Zero runs produced a wrong payout (0 wrong outcomes).

Five defect classes explain every failure. Each one is a contract defect, not a reasoning failure: in every case the needed fact existed in the desk and was missing from the contract.

  • D-1: A required input the model cannot know. get_policy takes name, while a ticket carries category (damaged_on_arrival, late_delivery, wrong_item). Every baseline run called get_policy with the category string and received {"error":"NOT_FOUND"} (F-06). Claude then tried search. Because search checks substrings of raw JSON, "damaged on arrival" matched POL-DOA-30's rule text and recovered T-1042 and T-3045, whereas "late delivery" matched nothing in POL-LATE-14, so T-2077 failed to find its policy in 3 of 3 runs (TR-1-T-2077.jsonl).
  • D-2: A closed set with no enum. update_ticket takes status as an unconstrained string and translates a hidden legacy table (open, pending, solved, escalated_l2). Across the 18 runs, Claude tried handed_to_human, pending_human, escalated, resolved, and closed, each answered with Error: invalid value, before guessing a word the handler accepted (F-02).
The tool said ok. The ticket stayed open. Baseline run T-3010, repetition 1, recorded 2026-11-15. Silent loop, 10 seconds.
Extended description

Four status calls in baseline transcript TR-2-T-3010.jsonl show why prose and HTTP 200 status codes cannot certify an outcome: handed_to_human, pending_human, and escalated each return Error: invalid value, while pending returns {"status":"ok"}.

Inside the baseline handler, a hidden legacy map translates pending back to open even though the model wrote HANDOFF TO HUMAN in its note. The harness record check (OUT-04) inspects the stored ticket at end_turn, sees open, escalates to needs_human, and points to the contract fix: a strict ["resolved", "needs_human"] enum.

This is the sharpest lesson in the packet. In all 12 safe stops (11 using pending and one using open), update_ticket returned {"status":"ok"}, Claude wrote a clear human-handoff note, and the legacy table mapped the status back to open. Without the harness checking the record at end_turn, all 12 runs would have looked successful in the transcript while sitting open in an unmonitored queue.

  • D-3: Overlapping tools with divergent rules. issue_refund and create_refund_v2 both issue refunds; issue_refund skips operator approval and idempotency (F-05). update_ticket and close_ticket both change ticket status (F-04). On 2026-11-15, all 6 live baseline payouts used create_refund_v2 with integer cents because its description says Use instead of issue_refund. Safety rested on one sentence in one competing tool.
  • D-4: Results sized for the endpoint, not the decision. get_order dumps the order, the customer's email, phone, and street address, full order history, 14 shipping scans, fraud_review, and margin_cents (F-07). Baseline runs averaged 3,787 characters of tool results and 24,139 uncached input tokens, copying synthetic contact data into context on every turn.
  • D-5: Errors that report without directing. Baseline handlers return Error: invalid value, {"error":"NOT_FOUND"}, 500 Internal Server Error, and a raw TypeError stack trace (F-06). None states the offending field, a valid value, whether a retry can succeed, or what to call next. Return a tool_result with is_error: true for a failed client tool call. Claude may try again, so set a retry limit in your loop rather than assuming a fixed number of retries (Handle tool calls). The 18 baseline runs recorded 59 tool errors.

One more defect stays invisible in the transcript: get_ticket mutates assignee and last_viewed_at on every read (F-01), logging a write by a read tool on 18 of 18 runs.

LAB-001: diagnose from transcripts (20 minutes)

In the hands-on labs below, run LAB-001 ($0.00, offline). You run pnpm trace on three unedited 2026-11-15 Claude transcripts (TR-1-T-2077.jsonl, TR-2-T-3010.jsonl, TR-3-T-1042.jsonl), map each failure to D-1 through D-5 or read-that-writes with its step number and 12-character call id, and state the contract field that removes it.

Repair 1: responsibilities and names

Start with capability ownership. List the distinct capabilities your tasks need: ticket.read, order.read, policy.read, customer.read, payout.refund, payout.store_credit, ticket.status, and ticket.notes. Give each needed capability one owning tool (C-02), and leave customer.read unowned because no task needs customer contact data.

Diagram: nine baseline tools on the left (search, get_ticket, get_order, get_policy, issue_refund, create_refund_v2, add_credit, update_ticket, close_ticket) connect to eight capabilities in the middle; capabilities claimed by more than one tool are amber. Three redesigned tools on the right connect in mint: desk_get_ticket_context owns ticket.read, order.read, and policy.read; payouts_issue owns payout.refund and payout.store_credit; desk_update_ticket owns ticket.status and ticket.notes. customer.read has no owner after the redesign.
Nine overlapping tools become three capability owners, with unused customer contact reads left unowned.
Extended description

On the left, eight of the nine baseline wrappers (search, get_ticket, get_order, get_policy, issue_refund, create_refund_v2, update_ticket, close_ticket) collide across shared capabilities or bundle unneeded fields; only add_credit owns one capability alone.

On the right, three redesigned tools partition the desk cleanly: desk_get_ticket_context owns ticket.read, order.read, and policy.read; payouts_issue owns payout.refund and payout.store_credit; and desk_update_ticket owns ticket.status and ticket.notes, while customer.read stays unowned because no support rule uses customer contact data.

Two forces pull in opposite directions when you group endpoints into tools:

  • Consolidate what the agent always chains. Every support ticket needs the ticket, its order, and its policy before any decision, so the redesign folds all three into one read tool, desk_get_ticket_context. Consolidating chained reads removes D-1 by looking up the policy from ticket.category in code, matching Anthropic's advice to build workflow tools rather than thin wrappers (Writing effective tools for AI agents) and to group related operations cleanly (Define tools).
  • Split across risk classes. Consolidate within a risk class, never across one. Putting reads and payouts behind a single action parameter makes read-only enforcement and operator approval depend on inspecting arguments rather than tool identity. Refunds and store credit share a risk class (both move money and require approval), so they share payouts_issue via payout_type. Updating ticket status moves no money, so desk_update_ticket stays separate.

Namespace every tool resource-first (desk_get_ticket_context, payouts_issue, desk_update_ticket) within the API's ^[a-zA-Z0-9_-]{1,128}$ name syntax (C-01, Define tools). Anthropic notes that prefix versus suffix namespacing has measurable, model-dependent effects, so keep one consistent scheme and test it on your tasks (Writing effective tools for AI agents).

Write descriptions as the primary specification. Claude's tool documentation states of descriptions: "This is by far the most important factor in tool performance." and advises, "Aim for at least 3-4 sentences for each tool description, more if the tool is complex." (Define tools). Lint rule C-03 requires at least three sentences and an explicit when-not-to-use boundary:

payouts_issuets
name: 'payouts_issue',description:  "Move money for one ticket: a refund to the original payment method, or store credit, as the ticket's policy requires. " +  'This changes the system of record, and a human operator must approve the exact payout before it happens. ' +  "Use it only after desk_get_ticket_context, with payout_type equal to the policy's payout_type and amount_cents no larger than the policy's max_payout_cents. " +  'Do not use it for replacements, for amounts over the limit, or to correct an earlier payout; hand those to a human with desk_update_ticket. ' +  'Repeating the same call is safe: it returns the original payout instead of paying twice.',strict: true,

Repair 2: inputs that cannot be misread

Every input parameter should state what it is, its unit or format, and where the value comes from without consulting another document. Anthropic's concrete example is unambiguous naming: "instead of a parameter named user , try a parameter named user_id ." (Writing effective tools for AI agents). Apply five rules across every schema:

  • Specific parameter names (C-04). ticket_id and order_id, never id, ticket, or order.
  • Integer cents (C-05). amount_cents with type: "integer" and 4800 means $48.00 in its description.
  • Closed sets as enums (C-06). status: ["resolved", "needs_human"] and payout_type: ["refund", "store_credit"], using the desk's real vocabulary.
  • Identifier patterns (C-06). ^T-[0-9]{4}$ and ^ORD-[0-9]{4}$.
  • Explanation, never reasoning. payouts_issue asks for a one-sentence explanation for the operator. Claude's tool guide warns, "Ask for an explanation, not reasoning." (Define tools), because parameters requesting internal reasoning can trigger a reasoning_extraction refusal (billed when emitted before output tokens since 2026-09-24, Release notes).

Enable strict tool use (strict: true on the tool definition and additionalProperties: false on input_schema, C-07) so grammar-constrained sampling guarantees schema-valid inputs: "Strict tool use is available on every model and eligible for HIPAA readiness." (Strict tool use).

input_schemats
input_schema: {  type: 'object',  properties: {    ticket_id: TICKET_ID,    response_format: {      type: 'string',      enum: ['concise', 'detailed'],      description: "'concise' for deciding (default). 'detailed' only when you need shipping events or earlier notes.",    },  },  required: ['ticket_id'],  additionalProperties: false,},

Two related request features carry sharp rules:

  • input_examples. Optional sample inputs add input tokens and are unsupported on server tools. Count tokens on your actual request, and validate each example against input_schema before sending it (Define tools). On 2026-11-15, passing integer 1042 for ticket_id returned 400 (tools.0.custom: Example at index 0 is invalid: 1042 is not of type "string". Each example must match the tool's \input_schema\.). Always use synthetic IDs outside your evaluation set (T-9001, ORD-9001) so examples cannot leak task answers.
  • tool_choice on Sonnet 5.5 (F-08). On claude-sonnet-5-5 (as well as Opus 5.5, Fable 5.2 and 5.1, and Mythos 5.2 and 5.1), "any and tool return a 400 error." (Define tools); on 2026-11-15 both returned tool_choice: type "tool" and "any" are not supported for this model. Use auto with strict tools, and remember that changing tool_choice mid-conversation invalidates cached message blocks.

Repair 3: errors the model can act on

A tool error is an instruction to a model that will act on the next turn. Return execution failures with is_error: true on the tool_result block and "include what went wrong and what Claude should try next" (Handle tool calls). Over MCP (2026-07-28), tool execution failures follow the identical pattern: "They are reported in tool results with isError: true" rather than as protocol-level JSON-RPC errors so the model can recover (MCP Tools, Guide 04).

Fields of an actionable structured tool error enforced by fail() in src/contracts/result.ts.
FieldQuestion it answersExample value
codeWhich stable failure class occurred?AMOUNT_OVER_POLICY_LIMIT
field and expectedWhich input caused it, and what value is valid?amount_cents, at most 6000
retryableCan repeating or adjusting this call succeed?false
next_stepWhich action or tool should the model take next?Do not retry with a different amount. Call desk_update_ticket with status 'needs_human'...
Diagram: on the left, an amber box of four baseline errors: Error: invalid value; error NOT_FOUND; 500 Internal Server Error; TypeError: Cannot read properties of undefined, with the note that none names the input, a valid value, retry advice, or a next step. On the right, a mint box shows the redesigned JSON error with code AMOUNT_OVER_POLICY_LIMIT, message, field amount_cents, expected at most 6000, retryable false, and a next_step that says not to retry and to call desk_update_ticket with status needs_human.
Errors the model can act on: four opaque baseline strings versus one structured recovery contract.
Extended description

Baseline handlers return Error: invalid value, {"error":"NOT_FOUND"}, 500 Internal Server Error, and an uncaught TypeError stack trace, omitting the input field, valid values, retry guidance, and next action.

The redesigned payouts_issue error on T-3022 returns code: "AMOUNT_OVER_POLICY_LIMIT", field: "amount_cents", expected: "at most 6000", retryable: false, and a next_step naming desk_update_ticket with status: "needs_human", ending retry loops in one turn.

fail() in src/contracts/result.ts enforces this shape (C-11). Three habits prevent retry loops:

  1. Say Do not retry when retrying cannot succeed. Policy limits, expired windows, out-of-stock SKUs, and operator denials will fail the same way on every attempt; retryable: false plus desk_update_ticket with needs_human ends the turn cleanly.
  2. Supply the valid value when the server already has it. When order_id does not match the ticket's order, ORDER_NOT_ON_TICKET puts ticket.order_id in expected.
  3. Never echo upstream tracebacks or raw vendor codes. Map internal exceptions and upstream strings (E_STOCK_0 in Transfer: wrap a new upstream API) into domain error codes.

Repair 4: results sized for the next decision

"Tool definitions and accumulated tool_result blocks consume your context window." (Manage tool context). Every character a read tool returns is billed again on every subsequent model call in the run unless context editing removes it. Anthropic recommends returning high-signal fields, "exposing a simple response_format enum parameter in your tool, allowing your agent to control whether tools return" concise or detailed responses, and adding pagination or explicit truncation limits; as a reference point, "For Claude Code, we restrict tool responses to 25,000 tokens by default." (Writing effective tools for AI agents).

On the six lab tickets, desk_get_ticket_context in concise mode returns 704 to 925 characters (932 mean result characters across entire runs, down from 3,787 in the baseline): the ticket, order line items in integer cents, policy rules, and precomputed checks.within_window so the model never performs calendar math. detailed adds the 5 latest shipping events (stating that 5 of 14 are shown) and prior staff notes. Customer email, phone, street address, fraud_review, and margin_cents never appear (C-10).

Concise resultjson
{"ticket":{"ticket_id":"T-3031","category":"damaged_on_arrival","status":"open","opened_on":"2026-10-05","customer_message":"The zipper pull on my sleeping bag came off the first time I used it. ..."}, "order":{"order_id":"ORD-7304","delivered_on":"2026-08-20","items":[{"sku":"BAG-20F","name":"20-degree sleeping bag","unit_price_cents":18900,"qty":1}],"total_cents":18900,"refunded_cents":0}, "policy":{"policy_id":"POL-DOA-30","payout_type":"refund","max_payout_cents":6000,"window_days":30,"rule":"Damaged on arrival and reported within 30 days of delivery: ..."}, "checks":{"days_from_delivery_to_report":46,"within_window":false}}

Repair 5: side effects in the contract

A contract must declare its side effects honestly, and your handler code must enforce that declaration:

  • Reads change nothing (C-08, C-09). Every contract declares effect: 'read' or effect: 'write'. Test C-09 invokes every read tool against a clean desk and fails if any state or journal entry changes.
  • Writes require exact-action approval (C-14). payouts_issue calls requireApproval with a fingerprint binding ticket, order, payout type, and amount_cents before mutating state (Guide 01, Guide 03).
  • Writes are idempotent (C-12). Deriving payout:<ticket>:<order>:<type>:<amount> lets a repeated call after a network timeout return the original payout with already_existed: true.
  • Policy is enforced in handler code (C-13). payouts_issue re-reads the policy and rejects wrong payout types, expired windows, and over-limit amounts before asking an operator or writing to the ledger.

Mapping side-effect contracts to MCP ToolAnnotations

When you expose these same contracts through an MCP server (Guide 04), the 2026-07-28 MCP schema defines ToolAnnotations ("Optional annotations for the client.", MCP Schema): readOnlyHint (default false), destructiveHint (default true, meaningful when readOnlyHint is false), idempotentHint (default false), and openWorldHint (default true). In our redesigned library, desk_get_ticket_context maps to readOnlyHint: true, openWorldHint: false, while payouts_issue maps to readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: false. Treat annotations strictly as UI and routing hints, never as security enforcement: the MCP specification requires clients to "consider tool annotations to be untrusted unless they come from trusted servers." (MCP Tools). Approval, idempotency, and policy checks stay in server code.

LAB-002: redesign the library (40 minutes)

In LAB-002 (hands-on labs), replace src/contracts/redesigned.ts (which starts as a copy of the baseline) across steps R1 to R5 until all three offline gates pass ($0.00):

LAB-002 checksbash
pnpm lint:contracts --library redesignedpnpm testpnpm eval --library redesigned

On the untouched starter, pnpm test reports # pass 2 and # fail 8; after your redesign it reports # pass 10 and # fail 0, and pnpm eval --library redesigned passes 6 of 6.

Measure the redesign, then decide on discovery

Evaluate contracts against the system of record (OUT-01 to OUT-04) alongside call counts, errors, result characters, and token usage (LAB-003):

Live comparisonbash
pnpm eval --library baseline --model claude --reps 3 --out .runs/eval-baseline.jsonpnpm eval --library redesigned --model claude --reps 3 --out .runs/eval-redesigned.jsonpnpm compare .runs/eval-baseline.json .runs/eval-redesigned.json
Bar chart, amber for baseline and mint for redesigned, recorded 2026-11-15. Tasks passed of 18: 6 versus 18. Mean model calls per run: 7.4 versus 3.5. Mean uncached input tokens per run: 24,139 versus 9,422. Mean tool-result characters per run: 3,787 versus 932. Definition-only delta over no tools (includes the 286-token tool-use system prompt): 1,096 versus 1,949 tokens. Mean estimated run cost at 2026-11-15 list prices: $0.0590 uncached versus $0.0244 uncached ($0.0105 with automatic prompt caching across 12 runs).
Same tasks, same model, different contracts. Live: claude-sonnet-5-5, effort medium. Recorded 2026-11-15.
Extended description

Across 18 live runs per library (6 synthetic tickets times 3 repetitions on 2026-11-15 with claude-sonnet-5-5 at effort medium), passing runs rose from 6 to 18, mean model calls fell from 7.4 to 3.5, mean uncached input tokens fell from 24,139 to 9,422, and mean tool-result characters fell from 3,787 to 932.

Definition-only overhead (tool_choice: none) rose from +1,096 tokens on the nine terse wrappers to +1,949 tokens on the three strict contracts (including Anthropic's 286-token tool-use system prompt), yet mean estimated run cost dropped from $0.0590 uncached to $0.0244 uncached and $0.0105 with --cache auto (12 runs).

Cost in two views: uncached tokens and automatic prompt caching

Look closely at the definition-only row first (pnpm defs:cost, measured with tool_choice: { type: "none" } on 2026-11-15). On claude-sonnet-5-5, any request carrying tools adds Anthropic's 286-token tool-use system prompt (Pricing). Over a 17-token no-tools prompt, the nine baseline definitions added +1,096 tokens (1,113 total, or 810 pure schema tokens after the 286-token system prompt), while the three redesigned definitions added +1,949 tokens (1,966 total, 1,663 pure schema tokens). Good contracts cost more definition tokens up front. The 39 percent uncached input-token ratio (9,422 versus 24,139) comes from cutting model calls from 7.4 to 3.5, cutting result characters from 3,787 to 932, and eliminating 59 error-probing turns.

Now view the same redesigned library under automatic prompt caching (pnpm eval --cache auto, which sets "Add a single cache_control field at the top level of your request.", Pricing; 512-token minimum on Sonnet 5.5, Prompt caching). At 2026-11-15 Sonnet 5.5 list prices ($2.00 base input, $2.50 5-minute cache write, $0.10 cache hit since 2026-10-07, and $10.00 output per million tokens, Models overview, Pricing), 18 uncached baseline runs cost $1.06 ($0.0590 mean) and 18 uncached redesigned runs cost $0.44 ($0.0244 mean). With --cache auto across 12 runs (6 tickets times 2 repetitions), mean redesigned cost dropped to $0.0105 ($0.0072 on warm second repetitions; 1,701 mean cache-write tokens and 7,707 mean cache-read tokens). Always verify usage fields in your logs: "if both cache_creation_input_tokens and cache_read_input_tokens are 0, the prompt was not cached" (Prompt caching), and editing any tool definition "invalidates the entire cache".

When to add tool discovery

Manage tool context recommends a clear sequence for "Stable tool definitions across many requests": turn on prompt caching first, add tool search at 10 or more tools (or above 10,000 definition tokens, as selection accuracy degrades past 30 to 50 tools), add context editing for long sessions (Guide 08), and reserve programmatic calling for repetitive multi-call loops. Tool search marks long-tail tools defer_loading: true (up to "10,000 tools with defer_loading: true for each request") alongside tool_search_tool_regex_20251119 (case-insensitive Python re.search patterns up to 200 chars) or tool_search_tool_bm25_20251119 (natural-language queries up to 500 chars), and both variants search "tool names, descriptions, argument names, and argument descriptions" (Tool search tool). Two request rules return 400 if broken: "Never set defer_loading: true on the tool search tool itself." (F-11, At least one tool must have defer_loading=false. All tools cannot be deferred.), and "A tool with defer_loading: true can't also carry cache_control: the API returns a 400." (F-10, Tool 'desk_probe_tool_01' cannot have both defer_loading=true and cache_control set. Tools with defer_loading cannot use prompt caching.). Top-level automatic cache_control works cleanly with deferred tools.

To test discovery on 2026-11-15, we added 45 well-formed tools from nine other Example Outfitters departments (inventory_*, shipping_*, vendors_*, hr_*, finance_*, and others) that no support ticket needs (12 live runs per condition, all 12/12 passing):

Discovery runsbash
pnpm eval --library redesigned --discovery catalog-loaded   --model claude --reps 2pnpm eval --library redesigned --discovery catalog-deferred --model claude --reps 2pnpm eval --library redesigned --discovery search-all       --model claude --reps 2pnpm eval --library redesigned --discovery search-all --search-variant bm25 --model claude --reps 2pnpm defs:cost
Discovery pays when definitions dominate. Live: claude-sonnet-5-5, effort medium. Recorded 2026-11-15. Silent loop, 10 seconds.
Extended description

Across six tool-loading conditions (18 runs on 3 desk tools; 12 runs on each 48-tool condition, 100% task pass rate), loading all 45 non-desk catalog tools raised definition delta to +8,781 tokens and mean input tokens to 33,332 ($0.0213 cached), while pinning the 3 desk tools and deferring the 45 catalog tools (catalog-deferred) held mean input tokens to 10,766 (+2,334 definition delta, $0.0113 cached, 0 search calls).

Deferring all 48 tools behind regex search cost 30,997 mean input tokens when catalog descriptions ended with Not for customer support tickets, refunds, or store credit., 14,842 tokens with department-only wording (6 domain-noun matches across name, description, argument name, and argument description), and 15,842 tokens under BM25 (25 search calls across 12 runs).

Measured definition deltas, search calls, input tokens, and estimated run costs across six discovery conditions (2026-11-15).
Condition (12 runs each)Definition deltaSearch callsMean uncached input tokensUncached mean USDCached mean USD (warm rep 2)
3 desk tools only (redesigned, 18 runs)+1,949n/a9,422$0.0244$0.0105 ($0.0072)
48 tools all loaded (catalog-loaded)+8,781n/a33,332$0.0722$0.0213 ($0.0098)
3 desk loaded + 45 catalog deferred (catalog-deferred, regex)+2,334010,766$0.0271$0.0113 ($0.0076)
All 48 deferred, catalog mentions tickets/refunds (search-all, regex)+6711230,997$0.0685not run
All 48 deferred, department-only wording (search-all, regex)not measured separately1214,842$0.0362not run
All 48 deferred, catalog mentions tickets/refunds (search-all, BM25)+6432515,842$0.0385not run

Three findings stand out, and the discovery decision holds in both the uncached and prompt-cached views:

  1. Keep core tools loaded and defer the long tail. "Keep your 3-5 most frequently used tools non-deferred so Claude can call them without searching first." (Tool search tool; echoed in Introducing advanced tool use, "Keep your three to five most-used tools always loaded, defer the rest."). In catalog-deferred, Claude made 0 search calls across 12 runs, cutting uncached input tokens from 33,332 to 10,766 and cached mean cost from $0.0213 to $0.0113 (-47%, and $0.0076 versus $0.0098 on warm repetitions).
  2. Regex search matches four fields, not descriptions alone (F-12). Deferring all 48 tools looked cheapest in defs:cost (+671 tokens) yet averaged 30,997 input tokens per run when each catalog description ended with Not for customer support tickets, refunds, or store credit. On 12 of 12 runs Claude searched ticket|order|policy|refund. Offline field attribution (attributeRegexQuery in src/contracts/catalog.ts) shows all 45 catalog tools matched: 41 on description alone and 4 on multiple fields. Rewriting the catalog to department-only wording (Use it only for inventory work.) dropped mean input tokens to 14,842, not all the way to 10,766, because ticket|order|policy|refund still matched 6 catalog tools across all four searched fields: inventory_list_low_stock (reorder point in description), inventory_get_reorder_plan (reorder in name and orders in description), shipping_void_label (refund of postage in description), vendors_create_purchase_order and vendors_list_purchase_orders (purchase_order in name and description), and vendors_record_delivery (purchase_order_id in arg_name and Purchase order id. in arg_description).
  3. BM25 caps returned tools at 5 by default, and negation boilerplate still hurts ranking. Switching search-all on mentions-desk to tool_search_tool_bm25_20251119 (--search-variant bm25; server-side ranking, so offline field attribution applies only to regex) lowered mean input tokens from 30,997 to 15,842 ($0.0385). Yet on 12 of 12 runs, Claude's first query (read support ticket or read ticket) returned five short read-only catalog tools (hr_get_employee, finance_get_tax_summary, vendors_get_vendor, stores_get_store_hours, it_get_device) whose concise descriptions had high term density on read, support, and tickets, missing desk_get_ticket_context and forcing a second broader BM25 search (25 search calls across 12 runs; "The response you receive contains server_tool_use blocks showing what ran and what came back.", How tool use works). catalog-deferred (10,766 tokens) beat search-all under every search variant.

When to add programmatic tool calling

Programmatic tool calling (code_execution_20260120 or code_execution_20260521 in allowed_callers) lets Claude run loops and filters over tool results inside a code-execution container so intermediate payloads stay out of model context (Programmatic tool calling). Anthropic's documentation reports sharp workload boundaries: "On a 75-tool project-management agent benchmark, enabling programmatic tool calling reduced billed input tokens by roughly 38 percent with no change in task accuracy.", and requests carrying 10 to 49 tools typically save 20 to 40 percent, whereas on τ²-bench (one or two sequential calls per turn) accuracy was unchanged and total cost rose roughly 8 percent: "Sequential single-call workflows do not benefit." (Earlier 2025-11-24 benchmarks in Introducing advanced tool use belong to multi-step research workloads, not 3-call support tickets.) Moreover, "Tools with strict: true are not supported with programmatic calling." Adopting it on Example Outfitters would forfeit strict grammar guarantees on payouts_issue and desk_update_ticket for a 3-step sequential workflow that docs show costs more. We therefore teach it as an explicit "do not adopt here" engineering decision; when a batch fan-out workload does fit (such as ASM-003's 400-payout reconciliation), measure billed input tokens with and without allowed_callers on read-only tools first.

LAB-003: measure and decide (20 minutes)

Follow LAB-003 in the hands-on labs to compare baseline and redesigned offline ($0.00), optionally run the live and --cache auto comparisons, and write a one-page decision record stating your shipping choice, discovery configuration, programmatic-calling decision, and sample limits.

Setup and runtime troubleshooting

Setup and runtime troubleshooting for Node.js, TypeScript type stripping, SDK 1.8.0, and prompt caching.
SymptomLikely causeFix
pnpm preflight fails on Node.js versionNode.js older than 22.18.0, which first ran .ts files without an experimental flag ("v22.18.0 Type stripping no longer emits an experimental warning.", Modules: TypeScript)Use Node.js 22.23.5 (.node-version), 24.22.3, or Active LTS 26.13.1
ERR_UNSUPPORTED_TYPESCRIPT_SYNTAXNon-erasable TypeScript syntax such as enum or constructor parameter propertiesUse string union types and explicit class fields; pnpm typecheck catches this via erasableSyntaxOnly
400 temperature: Extra inputs are not permittedSending temperature, top_p, or top_k on API version 2026-10-21 (@anthropic-ai/sdk 1.x)Remove sampling parameters and control thoroughness with output_config: { effort: "medium" }
Error: Could not resolve authentication method... Or for the "Authorization" header to be explicitly omittedProxy client using SDK 0.x's defaultHeaders: { 'x-api-key': null } on SDK 1.xUse defaultHeaders: { Authorization: null }: "If the proxy adds the credential and your client has none, v0 let you say so with either \"X-Api-Key\": null or Authorization: null in defaultHeaders. v1 accepts only Authorization: null." (Migrating to API version 2026-10-21)
400 when adding input_examplesExample payload violates input_schema (1042 instead of "T-9001")Validate every example against input_schema and use IDs outside the evaluation set
Both cache_creation_input_tokens and cache_read_input_tokens stay 0 with --cache autoPrompt is shorter than Sonnet 5.5's 512-token minimum, or tools changed between turnsKeep tools byte-identical across turns and verify total prompt exceeds 512 tokens
Transcript looks clean, yet pnpm eval scores safe_stop or wrong_outcomeTool accepted an ambiguous value (pending) or moved the wrong unit (48 cents)Inspect pnpm trace <jsonl> alongside OUT-01 to OUT-04 in src/eval/outcome.ts
Scripted double behaves differently from ClaudeExpected: the double executes five fixed rules in src/model/scripted.tsUse the double for $0.00 structural regression checks and --model claude for model evidence

Quick reference

Design rules

  1. One owner per capability (C-02). Consolidate reads the agent always chains; split writes across risk classes.
  2. Resource-first names (C-01). desk_get_ticket_context, payouts_issue, desk_update_ticket.
  3. 3+ sentence descriptions (C-03). State what it does, when to use it, when not to use it (in its own domain's vocabulary), units, limits, and side effects.
  4. Unambiguous strict inputs (C-04 to C-07). Specific *_id names with regex patterns, integer *_cents, closed enum sets, explanation instead of reasoning, strict: true, additionalProperties: false.
  5. Decision-sized results (C-10). Under 2,000 characters on read tools, response_format (concise or detailed) for optional context, zero unneeded contact or margin fields.
  6. Actionable errors (C-11). is_error: true with code, message, field, expected, retryable, and next_step; never raw stack traces or vendor codes.
  7. Enforced side effects (C-08, C-09, C-12 to C-14). Pure reads; approved, idempotent, policy-checked writes; treat MCP ToolAnnotations as untrusted hints.
  8. Outcome and two-view cost evaluation. Score the system of record (OUT-01 to OUT-04), report both uncached tokens and --cache auto dollars, and keep 3 to 5 frequent tools loaded when deferring a catalog.

Lab checks

Static contract rules (C-01 to C-08), handler behavior tests (C-09 to C-14), and outcome checks (OUT-01 to OUT-04).
CheckWhat it verifies
C-01Names are namespaced, lowercase, unique (^[a-z][a-z0-9]*(_[a-z0-9]+)+$)
C-02Each capability has at most one owning tool
C-03Descriptions have 3+ sentences and a when-not-to-use boundary
C-04Parameters are described and specifically named (not id, order, ticket, name)
C-05Money parameters are integer *_cents
C-06Closed sets declare enum; *_id parameters declare pattern
C-07strict: true and additionalProperties: false
C-08effect (read or write) is declared and matches the tool name
C-09Read tools leave desk state and the write journal untouched
C-10Read results stay under 2,000 characters and omit contact and margin fields
C-11Invalid calls return structured JSON errors with code, retryable, and next_step
C-12Repeating an approved payout with identical inputs pays once (already_existed: true)
C-13Policy limits, windows, and payout types are enforced in handler code
C-14Denied operator approval moves zero money and returns retryable: false
OUT-01 to OUT-04Payouts, final ticket status, matching approvals, and agent-recorded completion verified from desk state

Commands

Quick referencebash
pnpm preflight                                                      # check Node.js and environment ($0.00)pnpm typecheck                                                      # TypeScript 7 erasable-syntax check ($0.00)pnpm lint:contracts --library baseline                              # static rules C-01 to C-08 ($0.00)pnpm lint:contracts --library redesigned                            # must print 0 finding(s) ($0.00)pnpm test                                                           # C-09 to C-14, outcomes, characterization ($0.00)pnpm self-test                                                      # 12 planted checker faults ($0.00)pnpm gallery all                                                    # Failure gallery F-01 to F-13 ($0.00)pnpm trace fixtures/transcripts/TR-2-T-3010.jsonl                   # inspect a recorded transcript ($0.00)pnpm eval --library redesigned                                      # 6-ticket offline evaluation ($0.00)pnpm eval --assessment                                              # 4-task Part A assessment check ($0.00)pnpm eval --library redesigned --model claude --reps 3 --cache auto # optional live cached run (cents)pnpm eval --library redesigned --discovery catalog-deferred --model claude --reps 2pnpm eval --library redesigned --discovery search-all --search-variant bm25 --model claude --reps 2pnpm defs:cost                                                      # live definition token cost (cents)pnpm compare .runs/a.json .runs/b.json                              # side-by-side markdown table ($0.00)

Platform limits and version watch list (checked 2026-11-15)

Verified platform limits and version watch list (checked 2026-11-15).
ItemVerified value on 2026-11-15Watch on upgrade
Strict tool caps (Structured outputs)20 strict tools, 24 optional params, 16 union params, 24h grammar cacheRe-probe 20 vs 21 strict tools if compilation limits expand
tool_choice on claude-sonnet-5-5 (Define tools)auto and none supported; any and tool return HTTP 400Check Define tools model compatibility table on new model releases
Tool search (Tool search tool)Up to 10,000 deferred tools; regex pattern <= 200 chars; BM25 query <= 500 chars; default 5 refsKeep core tools non-deferred and audit server_tool_use queries
Prompt caching & pricing (Prompt caching, Pricing)512-token min on Sonnet 5.5; $2.00 in / $2.50 5m write / $0.10 hit / $10.00 out per MTok; 286-token tool system promptUpdate cost constants in scripts/eval.ts when list prices change
@anthropic-ai/sdk (@anthropic-ai/sdk, Migrating to API version 2026-10-21)1.8.0 (Apache-2.0, default API 2026-10-21; temperature/top_p/top_k rejected)Keep Authorization: null on proxy clients and check role on MessageParam
Node.js (Node.js release index and Release WG schedule, Modules: TypeScript)22.23.5 pinned (Maintenance LTS, EOL 2027-04-30); verified on 24.22.3 and Active LTS 26.13.1Move .node-version to Node.js 26 LTS well before 2027-04-30

Lab 2: Redesign the library

Replace src/contracts/redesigned.ts (which starts as a copy of the baseline) with three strict contracts (desk_get_ticket_context, payouts_issue, desk_update_ticket) across repairs R1 to R5 until static lint, all ten handler and outcome tests, and the six-ticket offline evaluation pass at $0.00. All step outputs below were recorded offline on 2026-11-15 on Node.js v22.23.5 and pnpm 12.21.0 at $0.00 (Offline: scripted model double, not Claude. Synthetic data. Simulated services.) inside /tmp/tool-contracts-work; your paths start where you unpacked tool-contracts-labs.zip.

Acceptance checks

  • pnpm lint:contracts --library redesigned prints 0 finding(s) across rules C-01 to C-08
  • pnpm test passes 10 of 10 checks covering C-09 to C-14 and OUT-01 to OUT-04
  • pnpm eval --library redesigned passes 6 of 6 tickets with 0 tool errors and 0 reads that wrote
  1. Lint the redesigned contract file

    $ pnpm lint:contracts --library redesigned

    Initial: Before editing src/contracts/redesigned.ts

    $ node scripts/lint.ts --library redesigned Contract lint: redesigned (9 tools) FAIL C-01 Names are namespaced: <resource>_<verb>[_<object>], lowercase, and unique (9) FAIL C-02 Each capability has exactly one owning tool (no overlapping responsibilities) (7) FAIL C-03 Descriptions say what the tool does, when to use it, and when not to (3+ sentences) (17) FAIL C-04 Every parameter is described and specifically named (no bare id, query, amount, type, name, fields) (32) FAIL C-05 Money is integer cents in a parameter whose name ends in _cents (4) FAIL C-06 Closed sets use enum; *_id parameters carry a pattern (3) FAIL C-07 Strict schemas: strict true, additionalProperties false, every property listed (18) FAIL C-08 Effects are declared: read tools use read verbs, write tools say what they change (4) Findings:

    Final: After repairs R1 to R5

    $ node scripts/lint.ts --library redesigned Contract lint: redesigned (3 tools) PASS C-01 Names are namespaced: <resource>_<verb>[_<object>], lowercase, and unique PASS C-02 Each capability has exactly one owning tool (no overlapping responsibilities) PASS C-03 Descriptions say what the tool does, when to use it, and when not to (3+ sentences) PASS C-04 Every parameter is described and specifically named (no bare id, query, amount, type, name, fields) PASS C-05 Money is integer cents in a parameter whose name ends in _cents PASS C-06 Closed sets use enum; *_id parameters carry a pattern PASS C-07 Strict schemas: strict true, additionalProperties false, every property listed PASS C-08 Effects are declared: read tools use read verbs, write tools say what they change 0 finding(s).
  2. Run the handler behavior and outcome test suite

    $ pnpm test

    Initial: Before editing src/contracts/redesigned.ts

    ok 1 - baseline fails every lint rule ok 2 - baseline reproduces the unit trap and the wrong-tool substitution with the double not ok 3 - C-01 to C-08: the contract lint is clean not ok 4 - C-09: read tools change nothing in the system of record not ok 5 - C-10: read results stay under budget and carry no customer contact or internal data not ok 6 - C-11: every tool returns structured, actionable errors for an unknown ticket not ok 7 - C-12: repeating the same payout call pays once not ok 8 - C-13: the policy limit is enforced in code, with a next step that hands off not ok 9 - C-14: a denied approval moves no money not ok 10 - OUT-01 to OUT-04: all six tasks pass with the redesigned library # tests 10 # pass 2 # fail 8

    Final: After repairs R1 to R5

    ok 1 - baseline fails every lint rule ok 2 - baseline reproduces the unit trap and the wrong-tool substitution with the double ok 3 - C-01 to C-08: the contract lint is clean ok 4 - C-09: read tools change nothing in the system of record ok 5 - C-10: read results stay under budget and carry no customer contact or internal data ok 6 - C-11: every tool returns structured, actionable errors for an unknown ticket ok 7 - C-12: repeating the same payout call pays once ok 8 - C-13: the policy limit is enforced in code, with a next step that hands off ok 9 - C-14: a denied approval moves no money ok 10 - OUT-01 to OUT-04: all six tasks pass with the redesigned library # tests 10 # pass 10 # fail 0
  3. Evaluate all six support tickets against the desk record

    $ pnpm eval --library redesigned
    $ node scripts/eval.ts --library redesigned Offline: scripted model double, not Claude. Synthetic data. Simulated services. Condition redesigned:scripted: 3 tool definitions (4119 characters) ticket rep verdict calls tools errors in_tok result_chars stop T-1042 1 pass 4 3 0 0 1083 end_turn T-2077 1 pass 4 3 0 0 998 end_turn T-3010 1 pass 3 2 0 0 749 end_turn T-3022 1 pass 3 2 0 0 867 end_turn T-3031 1 pass 3 2 0 0 895 end_turn T-3045 1 pass 4 3 0 0 1002 end_turn Summary: 6/6 pass, 0 safe stop, 0 wrong outcome; mean 3.5 model calls, 2.5 tool calls; 0 tool errors; mean 932 result chars per run
  4. Reproduce the baseline pending status trap

    $ pnpm gallery f-02
    $ node scripts/gallery.ts f-02 F-02 [Reproducible] update_ticket rejects "escalated" and maps "pending" silently back to "open" Evidence: status="escalated" -> Error: invalid value; status="pending" -> {"status":"ok"}, stored ticket.status="open"
  5. Verify all 12 planted checker faults

    $ pnpm self-test
    $ node scripts/self-test.ts PASS FLT-00: clean fixture passes C-01..C-08 with 0 findings PASS FLT-C-01: planted fault triggers C-01 independently PASS FLT-C-02: planted fault triggers C-02 independently PASS FLT-C-03: planted fault triggers C-03 independently PASS FLT-C-04: planted fault triggers C-04 independently PASS FLT-C-05: planted fault triggers C-05 independently PASS FLT-C-06: planted fault triggers C-06 independently PASS FLT-C-07: planted fault triggers C-07 independently PASS FLT-C-08: planted fault triggers C-08 independently PASS FLT-OUT: checkOutcome catches planted payout, shipment, status, approval, and read-write faults PASS FLT-STEP-0: runAgent returns steps=0 when model.create throws on call 1 PASS FLT-ATTR: attributeRegexQuery rejects empty input and separates name, description, arg_name, arg_description self-test: PASS (12 checks verified)
  6. Show the solution discussion

    Inspect the step-by-step checkpoints R1 through R5 or copy the finished reference into a separate scratch folder with node ../solutions/apply-solution.mjs final /tmp/contracts-final so you do not overwrite your own work.

    desk_get_ticket_context folds ticket, order, policy and precomputed checks.within_window into one pure read; payouts_issue enforces policy windows and caps, derives a deterministic idempotency key and calls requireApproval before mutating state; desk_update_ticket restricts status to resolved and needs_human and rejects RESOLVE_WITHOUT_ACTION.

More labs in this guide

  • Lab 1: Diagnose the baseline from transcripts

    Run static contract lint and the offline evaluation on the nine-tool baseline, then trace three unedited 2026-11-15 Claude transcripts (TR-1-T-2077, TR-2-T-3010, TR-3-T-1042) to map every failure to D-1 through D-5 or read-that-writes with its step number, 12-character call id, and contract fix.

  • Lab 3: Measure and decide

    Compare baseline and redesigned side by side offline ($0.00), optionally run a bounded 1-repetition live and --cache auto comparison on claude-sonnet-5-5, inspect regex field attribution with pnpm gallery f-12, and write a one-page engineering decision record.

Thirteen ways this breaks, and how you will know

Eight of these reproduce in the lab; the other five were observed in this guide's recorded runs. Learn the symptom first; it is what you will actually see.

  • F-01gallery f-01Reproducible

    Read tool mutates ticket assignment

    Symptom
    Opening a ticket with get_ticket reassigns ticket.assignee to "agent" and updates last_viewed_at, recording a write on all 18 of 18 baseline runs.
    Diagnosis
    Thin wrappers around legacy helpdesk endpoints inherit implicit "claim on view" mutations, turning a harmless read retry into a queue state change.
    Fix
    Remove ctx.desk.mutate from read handlers and declare effect: 'read' on desk_get_ticket_context.
    Verify
    "pnpm gallery f-01" shows the baseline journal entry; test C-09 in pnpm test confirms zero mutations across all redesigned reads.
  • F-02gallery f-02Reproducible

    The pending status trap

    Symptom
    update_ticket rejects escalated, handed_to_human, and pending_human with "Error: invalid value", then returns {"status":"ok"} for status: "pending" while leaving ticket.status set to "open" (11 of 12 live safe stops).
    Diagnosis
    status is typed as an unconstrained string backed by a hidden legacy map where pending translates to open.
    Fix
    Define status: { type: "string", enum: ["resolved", "needs_human"] } with strict: true on desk_update_ticket, and verify outcomes from the desk record rather than model prose.
    Verify
    "pnpm gallery f-02" reproduces the "ok" response and "open" stored status; C-06, C-07, and OUT-02 verify the enum and outcome check.
  • F-03gallery f-03Reproducible

    Dollar-versus-cent unit mismatch pays 48 cents

    Symptom
    On T-1042 ($48.00 tent pole), the scripted double calls issue_refund with amount: 48, paying $0.48 (48 cents) instead of $48.00 and failing OUT-01.
    Diagnosis
    issue_refund names its parameter amount: number with no unit in the name or description while add_credit takes dollars: number.
    Fix
    Name every currency field *_cents (amount_cents), set type: "integer", and include "4800 means $48.00" in the parameter description.
    Verify
    "pnpm gallery f-03" reproduces verdict=wrong_outcome on T-1042; C-05 and "pnpm eval --library redesigned" confirm 4800 cents.
  • F-04gallery f-04Reproducible

    close_ticket resolves an escalation after update_ticket fails

    Symptom
    On over-limit ticket T-3022, after update_ticket rejects escalated, the scripted double switches to close_ticket and marks the escalation resolved with zero payout (wrong_outcome).
    Diagnosis
    Two tools (update_ticket and close_ticket) share ticket.status, and close_ticket sets resolved unconditionally without checking whether the policy's action occurred.
    Fix
    Give ticket.status a single owner (desk_update_ticket) and return RESOLVE_WITHOUT_ACTION (expected: "needs_human") when resolved is requested with no recorded payout or replacement.
    Verify
    "pnpm gallery f-04" reproduces the false resolution; C-02 and pnpm test verify single ownership and RESOLVE_WITHOUT_ACTION.
  • F-05gallery f-05Reproducible

    Overlapping refund wrapper bypasses approval and idempotency

    Symptom
    Calling issue_refund twice under a denying approver still records two unapproved $48.00 refunds (9600 cents total, approved: false, idempotency_key: null).
    Diagnosis
    Legacy issue_refund remained in the tool list alongside create_refund_v2, leaving an unguarded write path one tool-selection mistake away.
    Fix
    Retire issue_refund and add_credit; consolidate money movement into payouts_issue gated by requireApproval and deterministic key payout:<ticket>:<order>:<type>:<amount>.
    Verify
    "pnpm gallery f-05" reproduces the double unapproved write; C-12 and C-14 prove idempotency and denied-approval safety.
  • F-06gallery f-06Reproducible

    Hidden policy IDs and raw stack traces

    Symptom
    get_policy {"name": "damaged_on_arrival"} returns {"error":"NOT_FOUND"} (21 chars), and get_order on an unknown ID returns "TypeError: Cannot read properties of undefined (reading 'lines')" with internal server file paths.
    Diagnosis
    get_policy expects POL-DOA-30 while asking for name, and get_order lets an uncaught exception serialize into tool_result.
    Fix
    Fold policy lookup by ticket.category into desk_get_ticket_context, and return structured fail() objects (is_error: true, code, field, expected, retryable, next_step) on every error path.
    Verify
    "pnpm gallery f-06" shows both raw baseline errors; C-11 tests invalid inputs across every redesigned contract.
  • F-07gallery f-07Reproducible

    Endpoint dump leaks contact data and internal margins

    Symptom
    get_order on ORD-5521 returns nearly 3,000 characters including @example.com email, +1-555-0110 phone, street address, 14 shipping events, fraud_review, and margin_cents.
    Diagnosis
    Returning the upstream ORM object exposes personal and internal fields no support rule uses and inflates every subsequent turn's prompt tokens.
    Fix
    Project only decision fields in desk_get_ticket_context (704 to 925 chars in concise mode), gate shipping history behind response_format: "detailed", and omit customer.read entirely.
    Verify
    "pnpm gallery f-07" flags the size and leaked fields; C-10 enforces < 2,000 characters and zero contact or margin keys across all tasks.
  • F-08Observed

    Forced tool_choice (any or tool) returns HTTP 400 on Sonnet 5.5

    Symptom
    Sending tool_choice: { type: "any" } or tool_choice: { type: "tool", name: "..." } to claude-sonnet-5-5 fails immediately with HTTP 400 invalid_request_error: tool_choice: type "tool" and "any" are not supported for this model.
    Diagnosis
    Claude Sonnet 5.5, Opus 5.5, Fable 5.2 and 5.1, and Mythos 5.2 and 5.1 reject forced tool selection (Claude API docs, Define tools).
    Fix
    Keep tool_choice at auto with strict: true tool schemas and clear system instructions, or use output_config.format structured outputs when you need a single JSON response rather than a tool loop.
    Verify
    Run "pnpm gallery f-08" to inspect the recorded 2026-11-15 error, and verify src/model/anthropic.ts sends tool_choice only for defs:cost (none).
  • F-09Observed

    21st strict tool or 25th optional parameter fails grammar compilation

    Symptom
    Marking a 21st tool strict: true returns HTTP 400: "Too many strict tools (21). The maximum number of strict tools supported is 20. Try reducing the number of tools marked as strict." Adding a 25th optional parameter across strict schemas returns HTTP 400: "Schemas contains too many optional parameters (25), which would make grammar compilation inefficient. Reduce the number of optional parameters in your tool schemas (limit: 24)."
    Diagnosis
    Grammar compilation enforces per-request caps of 20 strict tools, 24 optional parameters, and 16 union parameters (Claude API docs, Structured outputs).
    Fix
    Reserve strict: true for write tools and high-risk schemas, keep required parameters required unless genuinely optional, and leave low-risk read catalog tools non-strict.
    Verify
    "pnpm gallery f-09" prints both recorded 2026-11-15 400 messages; catalog.ts keeps the 45 non-desk catalog tools non-strict so 48-tool runs stay at 3 strict tools.
  • F-10Observed

    cache_control on a defer_loading: true tool returns HTTP 400

    Symptom
    Attaching cache_control: { type: "ephemeral" } directly to a tool definition that also sets defer_loading: true returns HTTP 400: "Tool 'desk_probe_tool_01' cannot have both defer_loading=true and cache_control set. Tools with defer_loading cannot use prompt caching."
    Diagnosis
    Deferred definitions are withheld from the initial prompt prefix, so block-level cache_control on a deferred tool entry is invalid (Claude API docs, Tool search tool).
    Fix
    Remove cache_control from deferred tool entries and enable automatic prompt caching with a single top-level cache_control: { type: "ephemeral" } field on the messages.create request (--cache auto).
    Verify
    "pnpm gallery f-10" prints the recorded 400 error; "pnpm eval --library redesigned --discovery catalog-deferred --cache auto" succeeds with cache writes and hits.
  • F-11Observed

    Deferring every tool including the search tool returns HTTP 400

    Symptom
    Setting defer_loading: true on every entry in tools returns HTTP 400: "At least one tool must have defer_loading=false. All tools cannot be deferred."
    Diagnosis
    When every tool is deferred, Claude has no loaded tool (not even the search tool) to discover the rest (Claude API docs, Tool search tool).
    Fix
    Never set defer_loading: true on tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119, and keep your 3 to 5 core workflow tools non-deferred.
    Verify
    "pnpm gallery f-11" prints the recorded 400 error; src/model/anthropic.ts leaves the search tool non-deferred in both catalog-deferred and search-all.
  • F-12gallery f-12Reproducible

    Negation boilerplate makes search-all load unrelated tools

    Symptom
    Putting "Not for customer support tickets, refunds, or store credit." in 45 non-desk catalog tools caused regex search-all (ticket|order|policy|refund) to match 45/45 catalog tools (30,997 mean input tokens), and caused BM25 search-all ("read support ticket") to miss desk_get_ticket_context on 12 of 12 initial queries (25 search calls total).
    Diagnosis
    Both regex and BM25 search tool names, descriptions, argument names, and argument descriptions (Claude API docs, Tool search tool). Naming another domain in a "when not to use" clause plants that domain's keywords into every deferred tool, and domain nouns like reorder or purchase_order_id still match order across 6 department-only tools.
    Fix
    Keep core tools non-deferred (catalog-deferred, 10,766 tokens) and write deferred descriptions in the positive vocabulary of their own domain ("Use it only for inventory work.").
    Verify
    "pnpm gallery f-12" runs attributeRegexQuery across both catalog variants and prints the 45/45 versus 6/45 field breakdown.
  • F-13Observed

    Beta tool_addition naming an undeclared tool returns tool_reference_unresolved

    Symptom
    Under anthropic-beta: inline-tools-2026-09-15, sending a mid-conversation role: "system" message with tool_addition referencing undeclared_desk_tool returns HTTP 400 (tool_reference_unresolved): "messages.1.content.0: tool_addition/tool_removal references unknown tool 'undeclared_desk_tool'". Placing that role: "system" message directly after an assistant text turn also returns 400 unless it follows a user message or a server-tool result.
    Diagnosis
    By-reference tool_addition can only surface tools already declared in the top-level tools array (typically with defer_loading: true), and system messages carry strict turn-order placement rules (Claude API docs, Mid-conversation system messages and tool changes).
    Fix
    Declare every candidate tool in tools up front with defer_loading: true, and inject role: "system" tool_addition blocks immediately after a user message.
    Verify
    "pnpm gallery f-13" prints the recorded 2026-11-15 tool_reference_unresolved error.

Transfer task and rubric

Wrap the simulated Example Parcel carrier API in one strict replacement-shipment write contract, diagnose an unedited baseline Claude transcript, and justify discovery and calling choices.

Analytic rubric: each criterion described at four performance levels, from Emerging to Exemplary
Criterion0 · Emerging1 · Developing2 · Proficient3 · Exemplary
R-A1 Responsibility and name (OBJ-002)No replacement tool added, or the tool claims an existing capabilityTool runs, but its name is verb-first or its description omits when-not-to-use or side-effect boundariesResource-first lowercase name, single capability shipment.replacement, and a 3+ sentence description stating when to use, when not to use, and what record it changesProficient, and rationale.md explains why replacement belongs in its own write tool rather than a third payout_type on payouts_issue
R-A2 Inputs and least data (OBJ-003)Model supplies a street address, customer contact field, or unvalidated free-form objectSpecific parameter names, but pattern, strict: true, or additionalProperties: false is missingticket_id and sku (or equivalent minimal identifiers) with regex patterns, strict: true, additionalProperties: false, and zero contact fieldsProficient, and the handler rejects a SKU that is not on the ticket's order with the ordered SKU named in expected
R-A3 Actionable errors (OBJ-004)Raw carrier codes (E_STOCK_0) or stack traces reach the modelSome carrier codes are mapped, but retryable, field, or next_step is missing or misleadingOut-of-stock, wrong-SKU, wrong-category, and non-automated-policy cases each return a stable code, honest retryable, and next_stepProficient, and E_DUP_REF is handled as an idempotent already_existed: true result with rationale.md explaining why a duplicate reference is not an agent error
R-A4 Decision-sized results (OBJ-004)Returns the raw carrier payloadTrimmed payload that still leaks label_zpl_base64, rates_considered, or address fieldsReturns shipment_id, sku, qty, eta_business_days, and already_existed in under 600 characters with zero contact or internal fieldsProficient, and rationale.md names which returned fields the agent needs to write the resolution note on desk_update_ticket
R-A5 Enforced side effects (OBJ-005)Ships twice on a retry, skips operator approval, or ships on a damaged-item ticketIdempotent and approved, but policy (payout_type, automated) is stated only in prose rather than enforced in handler codeDeclared write, gated by requireApproval, idempotent via a deterministic carrier ref, and checked against payout_type and automated in codeProficient, and the submission verifies that T-3010 still escalates on the main seed where POL-WRONG-30 has automated: false
R-A6 Interface rationale (OBJ-002 to OBJ-005)Missing or under 80 wordsSummarizes the code line by line without explaining design tradeoffs150 to 300 words covering model inputs, handler lookups, result trimming, error mapping, idempotency, and one rejected designProficient, and the rejected design is a plausible engineering alternative analyzed with its concrete token, safety, or schema cost
R-B1 Transcript diagnosis (OBJ-001)Restates the transcript narrative without defect classesNames defect classes without matching step numbers and 12-character call idsIdentifies at least three distinct defect classes in ASM-002-T-3031.jsonl with exact step and call id, and explains step 6's pending to open mapping and harness safe stopProficient, and ranks the top two contract repairs by exact steps saved on T-3031, comparing baseline's 7 steps with the 3-step redesigned pattern
R-C1 Architecture decision (OBJ-006)Gives opinions with neither measured numbers nor documented platform rulesCites either measured numbers or platform rules, but not both, or omits sample limitsBoth answers cite at least one 2026-11-15 measurement from "Measure the redesign, then decide on discovery" and one documented platform constraint, and state one limit of the evidenceProficient, and names the concrete pre-release experiment (including BM25 versus regex query inspection or allowed_callers read-only scoping) that could overturn the choice

Proficiency standard: Proficient or higher on every criterion. Proposed local standard, not an official certification.

Self-check

The full brief

In Part A (ASM-001), Example Outfitters enables automated replacements for wrong-item tickets on the assessment seed, backed by the simulated Example Parcel endpoint in src/services/carrier.ts. You add one declared write tool (shipment.replacement) to src/contracts/redesigned.ts without breaking the three existing contracts, keep customer contact data and base64 labels out of model inputs and results, map upstream carrier failures (E_STOCK_0, E_ADDR_17, E_DUP_REF, E_SVC) to actionable structured errors or idempotent outcomes, and submit a 150 to 300 word interface rationale.

In Part B (ASM-002), you trace fixtures/transcripts/ASM-002-T-3031.jsonl (Live: claude-sonnet-5-5, effort medium. Recorded 2026-11-15) with pnpm trace, classify every observed contract defect by step number and tool_use call id, explain why step 6 returned {"status":"ok"} while the run still ended in a harness safe stop, and rank the two contract changes that remove the most wasted steps.

In Part C (ASM-003), you write two 4 to 6 sentence engineering decisions from the guide's 2026-11-15 numbers and documented platform constraints: how to configure tool loading and search for a 120-tool multi-department support agent, and whether to adopt programmatic tool calling for a 400-payout nightly reconciliation job while keeping money-moving tools protected.

Passing requires Proficient (2) or higher on every criterion (R-A1 to R-A6, R-B1, R-C1) and no critical failure (CF-1 a write without operator approval or an idempotency key; CF-2 money accepted or returned as floating-point dollars; CF-3 a read tool that mutates desk state; CF-4 a tool input or result that exposes customer street address, email, phone, fraud score, or margin data). On 2026-11-15, the reference assessment solution passed 4 of 4 live tasks (3.8 mean model calls, 11,561 mean input tokens, $0.0284 mean run cost).

Teach it to a room

  • Facilitation guide and lab solutions

    A 125-minute workshop plan across LAB-001 to LAB-003 and the three-part transfer assessment, with hint ladders, live-room troubleshooting, and a lab solutions guide whose offline verification script checks every checkpoint and negative break test at $0.00.

  • Delivery for 12-person and 100-person rooms

    Concrete facilitation plans for an interactive 12-person cohort (with optional capped live comparisons) and a 100-person room running 100 percent offline with table leads and timed checkpoints. Newly authored planning designs; no cohort has run them yet.

  • Dry run and observation rubric

    A proposed 60-minute observed teach-back, live contract-repair drill, transcript diagnosis walk-through, and blind calibration scoring against four reference submissions. No dry run has been held yet.

  • Clearance record

    A per-instructor, per-version clearance template filled in only from an observed dry run and signed by an authorized reviewer. Blank: nobody has been cleared to teach this guide.

  • Evaluation plan and pass-rate template

    Pre-workshop diagnostic, lab checkpoint tracking, double-scored calibration protocol, learner privacy rules, and a cohort pass-rate report template with numerators and denominators. Blank until piloted.

  1. [1]Anthropic Engineering, Writing effective tools for AI agents, using AI agents (published 2025-09-11) · anthropic.comchecked Nov 15, 2026
  2. [2]Claude API docs, Define tools · platform.claude.comchecked Nov 15, 2026
  3. [3]Claude API docs, Strict tool use · platform.claude.comchecked Nov 15, 2026
  4. [4]Claude API docs, Structured outputs · platform.claude.comchecked Nov 15, 2026
  5. [5]Claude API docs, Handle tool calls · platform.claude.comchecked Nov 15, 2026
  6. [6]Claude API docs, How tool use works · platform.claude.comchecked Nov 15, 2026
  7. [7]Claude API docs, Tool search tool · platform.claude.comchecked Nov 15, 2026
  8. [8]Claude API docs, Programmatic tool calling · platform.claude.comchecked Nov 15, 2026
  9. [9]Claude API docs, Models overview · platform.claude.comchecked Nov 15, 2026
  10. [10]Claude API docs, Effort · platform.claude.comchecked Nov 15, 2026
  11. [11]npm registry, @anthropic-ai/sdk · npmjs.comchecked Nov 15, 2026
  12. [12]Node.js release index and Release WG schedule · nodejs.orgchecked Nov 15, 2026
  13. [13]Node.js 22 documentation, Modules: TypeScript · nodejs.orgchecked Nov 15, 2026
  14. [14]npm registry, pnpm, typescript, and @types/node · npmjs.comchecked Nov 15, 2026
  15. [15]Claude API docs, Manage tool context · platform.claude.comchecked Nov 15, 2026
  16. [16]Claude API docs, Prompt caching · platform.claude.comchecked Nov 15, 2026
  17. [17]Claude API docs, Mid-conversation system messages and tool changes · platform.claude.comchecked Nov 15, 2026
  18. [18]Claude API docs, Pricing · platform.claude.comchecked Nov 15, 2026
  19. [19]Claude API docs, Migrating to API version 2026-10-21 · platform.claude.comchecked Nov 15, 2026
  20. [20]Claude API docs, Release notes · platform.claude.comchecked Nov 15, 2026
  21. [21]Anthropic Engineering, Introducing advanced tool use on the Claude Developer Platform (published 2025-11-24) · anthropic.comchecked Nov 15, 2026
  22. [22]Model Context Protocol specification 2026-07-28, Server features: Tools · modelcontextprotocol.iochecked Nov 15, 2026
  23. [23]Model Context Protocol specification 2026-07-28, Schema reference (ToolAnnotations) · modelcontextprotocol.iochecked Nov 15, 2026
  24. [24]University of Hawai'i at Manoa Assessment and Curriculum Support Center, Creating and Using Rubrics · manoa.hawaii.educhecked Nov 15, 2026
  25. [25]Carnegie Mellon University Eberly Center, Creating and Using Rubrics · cmu.educhecked Nov 15, 2026
  26. [26]X Help Center, How to Post (and LinkedIn Help post limit check) · help.x.comchecked Nov 15, 2026

Independent work. Not affiliated with or endorsed by any vendor named in this guide.

Jordan Hudgens

Written by

Jordan Hudgens

Technical educator and AI engineering leader. 16 Udemy courses, 111K students, and curricula used by 250K+ learners.

Contact Jordan