AI-generated illustrationDesign Tool Contracts Agents Can Use Correctly
Turn a confusing tool library into contracts a model reads correctly, then prove the difference on the same tasks with the system of record, not the reply.
- Reading
- 69 min
- Workshop
- 125 min
- Stack
- TS · Claude API 2026-10-21 · @anthropic-ai/sdk 1.8.0 · Node.js 22 · pnpm 12
- Version
- 1.1.0
- Last verified
- Nov 16, 2026
01Outcomes
What you will build and be able to do
Each outcome has an observable criterion. The lab and the assessment check exactly these, nothing more and nothing less.
OBJ-001
Diagnose a tool-contract defect from a transcript
Diagnose a tool-contract defect from a transcript: name the defect class, cite the step and tool call, and name the contract change that removes it.
OBJ-002
Assign one owning tool to each capability
Assign one owning tool to each capability, with namespaced names and descriptions that say when to use the tool and when not to.
OBJ-003
Write input schemas a model cannot misread
Write input schemas a model cannot misread: specific names, integer cents, enums for closed sets, id patterns, and strict mode.
OBJ-004
Return decision-sized results and actionable errors
Return results sized for the next decision and errors that state the input at fault, a valid value, whether a retry can help, and the next step.
OBJ-005
Put side effects in the contract and enforce them in code
Put side effects in the contract: reads that change nothing, writes that are approved, idempotent, and checked against policy in code.
OBJ-006
Compare libraries on outcomes and cost, and decide on discovery
Compare two tool libraries on the same tasks with outcome checks and cost metrics, and decide from your own numbers whether tool discovery or programmatic calling is justified.
02Prerequisites and setup
Unpack the download and check your environment ($0.00)
- A working tool-use loop. You have built or read a loop where the model returns tool_use blocks and your code returns tool_result blocks. If not, start with Guide 01; this guide reuses its synthetic support desk and its bounded harness.Start with Guide 01
- TypeScript and JSON Schema. If you write Python, Go, or Rust, every contract rule translates directly.Read the contract primer
- Node.js 22.18.0 or later and pnpm 12. The download pins Node.js 22.23.5 (.node-version, Maintenance LTS with end of life on 2027-04-30, Node.js release index) and pnpm 12.21.0 (packageManager); the offline suite was also verified on Node.js 24.22.3 and Active LTS 26.13.1 on 2026-11-15.pnpm installation guide
- Optional Anthropic API key for live runs. Every lab and test runs offline at $0.00 with a scripted model double. Live comparisons in LAB-003 use @anthropic-ai/sdk 1.8.0 (Apache-2.0, default API version 2026-10-21) and cost well under a dollar per one-repetition comparison at 2026-11-15 list prices.
unzip -q tool-contracts-labs.zip -d tool-contracts-workcd tool-contracts-work/labs/startercorepack enablepnpm install --frozen-lockfilepnpm preflight03The system picture
Anatomy of a tool contract: three model-read parts, three code-enforced parts

Extended description
Left region, What the model reads, holds three stacked contract parts sent in the request: 1. Name (payouts_issue, domain first then verb, one name per capability); 2. Description (what it does, when to use it, when not to, units, policy caps, and state changes); and 3. Input schema (strict true, additionalProperties false, integer amount_cents, closed enums, and regex identifier patterns).
Right region, What your code enforces, holds three stacked runtime parts owned by handler code: 4. Result (only what the next decision needs, concise by default and detailed on request, with zero customer contact fields); 5. Error (stable code, field, expected valid value, honest retryable boolean, and next_step naming the recovery tool); and 6. Effect (declared read or write, pure reads, and approved, idempotent, policy-checked writes).
04Lesson
The contract is the interface
A tool contract is everything that defines how a tool can be used: its name, its description, its input schema, the shape of its results, the shape of its errors, and its effect on the world (whether it only reads, or changes something). The model sees the first three in the request and the next two as tool_result content. It never sees your handler code. Whatever the contract leaves unstated, the model has to guess.
That makes tool design an interface-design problem with an unusual consumer. A human developer reads your API reference once, asks a colleague, and remembers. A model reads the contract on every request, with no memory of the last integration and no one to ask. Anthropic's engineering guidance states the failure mode directly: "More tools don't always lead to better outcomes. A common error we've observed is tools that merely wrap existing software functionality or API endpoints" (Writing effective tools for AI agents).
The engineering decision this guide teaches is provider-independent: design each tool around one decision the agent has to make, and make every fact that decision needs explicit in the contract. Here the mechanism is Claude's direct tool-use API, and the same six contract parts appear when a server publishes tools over the Model Context Protocol (Guide 04).
Two rules keep the rest of the guide honest:
- The record decides success. A run passes only when the desk's own data shows the right payouts and the right ticket status. The model's final message is a claim. That is the rule Guide 01 builds into the harness, and it is why the baseline runs below recorded zero wrong payouts: the harness caught every bad finish.
- Measure on your tasks. Every number in this guide comes from six synthetic tickets, one model (
claude-sonnet-5-5), and one effort setting (medium). They show the size of the effects on this fixture, not a universal benchmark.
The lab: one desk, two libraries
The lab extends the synthetic support desk from Guide 01: Example Outfitters, a fictional outdoor retailer with tickets, orders, policies, synthetic @example.com customer profiles, shipping events, inventory, and a write journal. The journal records which tool made every change, so a test can prove that a read changed nothing.
The harness is Guide 01's repaired loop. It caps model calls at 10, stops after two identical failed calls, answers every tool_use with one tool_result placed first in the user content array ("the tool_result blocks must come FIRST in the content array. Any text must come AFTER all tool results.", Handle tool calls), and hands the ticket to a human on any stop that is not a verified finish. Only the tool library changes between runs.
| Ticket | Customer asks for | Policy | Correct outcome |
|---|---|---|---|
| T-1042 | $48.00 for a snapped tent pole | Damaged on arrival, $60.00 limit, 30 days | Refund 4800 cents, resolved |
| T-2077 | $15.00 credit for a late stove | Late delivery, store credit up to $15.00 | Store credit 1500 cents, resolved |
| T-3010 | The right daypack | Wrong item: replacement through a person | No payout, needs_human |
| T-3022 | $75.00 for a cracked cooler lid | Over the $60.00 limit | No payout, needs_human |
| T-3031 | $30.00, reported 46 days after delivery | Outside the 30-day window | No payout, needs_human |
| T-3045 | $12.50 for a headlamp strap | Damaged on arrival | Refund 1250 cents, resolved |
Unpack the download (tool-contracts-labs.zip) and check the starter offline ($0.00):
cd labs/starterpnpm install --frozen-lockfilepnpm preflightpnpm preflight checks your Node.js version and ends with Ready. plus an INFO line when no API key is set. To run the optional live commands in LAB-003, export your own key in your shell and hold effort constant:
export ANTHROPIC_API_KEY="<your-claude-console-key>" # never commit a keyexport AGENT_EFFORT=medium # explicit choice; Sonnet 5.5 defaults to highIn API version 2026-10-21 (@anthropic-ai/sdk 1.0.0 and later), sampling parameters temperature, top_p, and top_k return HTTP 400 (temperature: Extra inputs are not permitted, Migrating to API version 2026-10-21). Instead, the lab sets output_config: { effort: "medium" } as an engineering choice: Anthropic's effort guide notes high is the API default on Sonnet 5.5 and advises, "For agentic coding and multistep tool use, start with medium for well-specified tasks and move to high for harder or longer ones." (Effort). Hold effort constant whenever you compare two libraries.
Offline, every command uses a scripted model double (Offline: scripted model double, not Claude): a deterministic stand-in whose five rules sit at the top of src/model/scripted.ts. It copies an amount as the customer wrote it unless the parameter name or description says cents, and when an error gives no valid value it tries the next overlapping tool. Passing with the double proves your contracts removed those structural triggers; only live runs measure model behavior.
Diagnose the baseline
Read src/contracts/baseline.ts before running anything. It holds nine thin endpoint wrappers: search, get_ticket, get_order, get_policy, issue_refund, create_refund_v2, add_credit, update_ticket, and close_ticket. For example, get_policy asks for name while matching only internal IDs such as POL-DOA-30:
const getPolicy: ToolContract = { effect: 'read', capabilities: ['policy.read'], definition: { name: 'get_policy', description: 'Returns a policy.', input_schema: { type: 'object', properties: { name: { type: 'string' } }, required: ['name'] }, }, async handler(input, ctx) { const policy = ctx.desk.read().policies.find((p) => p.policy_id === str(input.name)); if (!policy) return { content: '{"error":"NOT_FOUND"}', isError: true }; return { content: JSON.stringify(policy), isError: false }; },};Run the static contract linter and the offline evaluation ($0.00):
pnpm lint:contracts --library baselinepnpm eval --library baselineThe linter fails all eight static rules (C-01 to C-08). Offline, the scripted double passes 2 of 6 tickets (4 wrong outcomes): it refunds 48 cents instead of $48.00 on T-1042 (F-03), and when update_ticket rejects escalated it falls back to close_ticket and marks three escalation tickets resolved (F-04). Claude failed differently, and its transcripts are more instructive.
What Claude did with the baseline
We ran claude-sonnet-5-5 at effort medium on the baseline library, 6 tickets times 3 repetitions (18 runs), through the direct Claude API on 2026-11-15 (Synthetic data. Simulated services.). Six of 18 runs passed. Twelve ended in a safe stop, where the harness set needs_human after checking the desk record. Zero runs produced a wrong payout (0 wrong outcomes).
Five defect classes explain every failure. Each one is a contract defect, not a reasoning failure: in every case the needed fact existed in the desk and was missing from the contract.
- D-1: A required input the model cannot know.
get_policytakesname, while a ticket carriescategory(damaged_on_arrival,late_delivery,wrong_item). Every baseline run calledget_policywith the category string and received{"error":"NOT_FOUND"}(F-06). Claude then triedsearch. Becausesearchchecks substrings of raw JSON,"damaged on arrival"matchedPOL-DOA-30's rule text and recoveredT-1042andT-3045, whereas"late delivery"matched nothing inPOL-LATE-14, soT-2077failed to find its policy in3of3runs (TR-1-T-2077.jsonl). - D-2: A closed set with no enum.
update_tickettakesstatusas an unconstrained string and translates a hidden legacy table (open,pending,solved,escalated_l2). Across the18runs, Claude triedhanded_to_human,pending_human,escalated,resolved, andclosed, each answered withError: invalid value, before guessing a word the handler accepted (F-02).
Extended description
Four status calls in baseline transcript TR-2-T-3010.jsonl show why prose and HTTP 200 status codes cannot certify an outcome: handed_to_human, pending_human, and escalated each return Error: invalid value, while pending returns {"status":"ok"}.
Inside the baseline handler, a hidden legacy map translates pending back to open even though the model wrote HANDOFF TO HUMAN in its note. The harness record check (OUT-04) inspects the stored ticket at end_turn, sees open, escalates to needs_human, and points to the contract fix: a strict ["resolved", "needs_human"] enum.
This is the sharpest lesson in the packet. In all 12 safe stops (11 using pending and one using open), update_ticket returned {"status":"ok"}, Claude wrote a clear human-handoff note, and the legacy table mapped the status back to open. Without the harness checking the record at end_turn, all 12 runs would have looked successful in the transcript while sitting open in an unmonitored queue.
- D-3: Overlapping tools with divergent rules.
issue_refundandcreate_refund_v2both issue refunds;issue_refundskips operator approval and idempotency (F-05).update_ticketandclose_ticketboth change ticket status (F-04). On2026-11-15, all6live baseline payouts usedcreate_refund_v2with integer cents because its description saysUse instead of issue_refund.Safety rested on one sentence in one competing tool. - D-4: Results sized for the endpoint, not the decision.
get_orderdumps the order, the customer's email, phone, and street address, full order history,14shipping scans,fraud_review, andmargin_cents(F-07). Baseline runs averaged3,787characters of tool results and24,139uncached input tokens, copying synthetic contact data into context on every turn. - D-5: Errors that report without directing. Baseline handlers return
Error: invalid value,{"error":"NOT_FOUND"},500 Internal Server Error, and a rawTypeErrorstack trace (F-06). None states the offending field, a valid value, whether a retry can succeed, or what to call next. Return atool_resultwithis_error: truefor a failed client tool call. Claude may try again, so set a retry limit in your loop rather than assuming a fixed number of retries (Handle tool calls). The18baseline runs recorded59tool errors.
One more defect stays invisible in the transcript: get_ticket mutates assignee and last_viewed_at on every read (F-01), logging a write by a read tool on 18 of 18 runs.
LAB-001: diagnose from transcripts (20 minutes)
In the hands-on labs below, run LAB-001 ($0.00, offline). You run pnpm trace on three unedited 2026-11-15 Claude transcripts (TR-1-T-2077.jsonl, TR-2-T-3010.jsonl, TR-3-T-1042.jsonl), map each failure to D-1 through D-5 or read-that-writes with its step number and 12-character call id, and state the contract field that removes it.
Repair 1: responsibilities and names
Start with capability ownership. List the distinct capabilities your tasks need: ticket.read, order.read, policy.read, customer.read, payout.refund, payout.store_credit, ticket.status, and ticket.notes. Give each needed capability one owning tool (C-02), and leave customer.read unowned because no task needs customer contact data.

Extended description
On the left, eight of the nine baseline wrappers (search, get_ticket, get_order, get_policy, issue_refund, create_refund_v2, update_ticket, close_ticket) collide across shared capabilities or bundle unneeded fields; only add_credit owns one capability alone.
On the right, three redesigned tools partition the desk cleanly: desk_get_ticket_context owns ticket.read, order.read, and policy.read; payouts_issue owns payout.refund and payout.store_credit; and desk_update_ticket owns ticket.status and ticket.notes, while customer.read stays unowned because no support rule uses customer contact data.
Two forces pull in opposite directions when you group endpoints into tools:
- Consolidate what the agent always chains. Every support ticket needs the ticket, its order, and its policy before any decision, so the redesign folds all three into one read tool,
desk_get_ticket_context. Consolidating chained reads removesD-1by looking up the policy fromticket.categoryin code, matching Anthropic's advice to build workflow tools rather than thin wrappers (Writing effective tools for AI agents) and to group related operations cleanly (Define tools). - Split across risk classes. Consolidate within a risk class, never across one. Putting reads and payouts behind a single
actionparameter makes read-only enforcement and operator approval depend on inspecting arguments rather than tool identity. Refunds and store credit share a risk class (both move money and require approval), so they sharepayouts_issueviapayout_type. Updating ticket status moves no money, sodesk_update_ticketstays separate.
Namespace every tool resource-first (desk_get_ticket_context, payouts_issue, desk_update_ticket) within the API's ^[a-zA-Z0-9_-]{1,128}$ name syntax (C-01, Define tools). Anthropic notes that prefix versus suffix namespacing has measurable, model-dependent effects, so keep one consistent scheme and test it on your tasks (Writing effective tools for AI agents).
Write descriptions as the primary specification. Claude's tool documentation states of descriptions: "This is by far the most important factor in tool performance." and advises, "Aim for at least 3-4 sentences for each tool description, more if the tool is complex." (Define tools). Lint rule C-03 requires at least three sentences and an explicit when-not-to-use boundary:
name: 'payouts_issue',description: "Move money for one ticket: a refund to the original payment method, or store credit, as the ticket's policy requires. " + 'This changes the system of record, and a human operator must approve the exact payout before it happens. ' + "Use it only after desk_get_ticket_context, with payout_type equal to the policy's payout_type and amount_cents no larger than the policy's max_payout_cents. " + 'Do not use it for replacements, for amounts over the limit, or to correct an earlier payout; hand those to a human with desk_update_ticket. ' + 'Repeating the same call is safe: it returns the original payout instead of paying twice.',strict: true,Repair 2: inputs that cannot be misread
Every input parameter should state what it is, its unit or format, and where the value comes from without consulting another document. Anthropic's concrete example is unambiguous naming: "instead of a parameter named user , try a parameter named user_id ." (Writing effective tools for AI agents). Apply five rules across every schema:
- Specific parameter names (
C-04).ticket_idandorder_id, neverid,ticket, ororder. - Integer cents (
C-05).amount_centswithtype: "integer"and4800 means $48.00in its description. - Closed sets as enums (
C-06).status: ["resolved", "needs_human"]andpayout_type: ["refund", "store_credit"], using the desk's real vocabulary. - Identifier patterns (
C-06).^T-[0-9]{4}$and^ORD-[0-9]{4}$. - Explanation, never reasoning.
payouts_issueasks for a one-sentenceexplanationfor the operator. Claude's tool guide warns, "Ask for an explanation, not reasoning." (Define tools), because parameters requesting internal reasoning can trigger areasoning_extractionrefusal (billed when emitted before output tokens since2026-09-24, Release notes).
Enable strict tool use (strict: true on the tool definition and additionalProperties: false on input_schema, C-07) so grammar-constrained sampling guarantees schema-valid inputs: "Strict tool use is available on every model and eligible for HIPAA readiness." (Strict tool use).
input_schema: { type: 'object', properties: { ticket_id: TICKET_ID, response_format: { type: 'string', enum: ['concise', 'detailed'], description: "'concise' for deciding (default). 'detailed' only when you need shipping events or earlier notes.", }, }, required: ['ticket_id'], additionalProperties: false,},Two related request features carry sharp rules:
input_examples. Optional sample inputs add input tokens and are unsupported on server tools. Count tokens on your actual request, and validate each example againstinput_schemabefore sending it (Define tools). On2026-11-15, passing integer1042forticket_idreturned400(tools.0.custom: Example at index 0 is invalid: 1042 is not of type "string". Each example must match the tool's \input_schema\.). Always use synthetic IDs outside your evaluation set (T-9001,ORD-9001) so examples cannot leak task answers.tool_choiceon Sonnet 5.5 (F-08). Onclaude-sonnet-5-5(as well as Opus 5.5, Fable 5.2 and 5.1, and Mythos 5.2 and 5.1), "anyandtoolreturn a 400 error." (Define tools); on2026-11-15both returnedtool_choice: type "tool" and "any" are not supported for this model.Useautowith strict tools, and remember that changingtool_choicemid-conversation invalidates cached message blocks.
Repair 3: errors the model can act on
A tool error is an instruction to a model that will act on the next turn. Return execution failures with is_error: true on the tool_result block and "include what went wrong and what Claude should try next" (Handle tool calls). Over MCP (2026-07-28), tool execution failures follow the identical pattern: "They are reported in tool results with isError: true" rather than as protocol-level JSON-RPC errors so the model can recover (MCP Tools, Guide 04).
| Field | Question it answers | Example value |
|---|---|---|
| code | Which stable failure class occurred? | AMOUNT_OVER_POLICY_LIMIT |
| field and expected | Which input caused it, and what value is valid? | amount_cents, at most 6000 |
| retryable | Can repeating or adjusting this call succeed? | false |
| next_step | Which action or tool should the model take next? | Do not retry with a different amount. Call desk_update_ticket with status 'needs_human'... |

Extended description
Baseline handlers return Error: invalid value, {"error":"NOT_FOUND"}, 500 Internal Server Error, and an uncaught TypeError stack trace, omitting the input field, valid values, retry guidance, and next action.
The redesigned payouts_issue error on T-3022 returns code: "AMOUNT_OVER_POLICY_LIMIT", field: "amount_cents", expected: "at most 6000", retryable: false, and a next_step naming desk_update_ticket with status: "needs_human", ending retry loops in one turn.
fail() in src/contracts/result.ts enforces this shape (C-11). Three habits prevent retry loops:
- Say
Do not retrywhen retrying cannot succeed. Policy limits, expired windows, out-of-stock SKUs, and operator denials will fail the same way on every attempt;retryable: falseplusdesk_update_ticketwithneeds_humanends the turn cleanly. - Supply the valid value when the server already has it. When
order_iddoes not match the ticket's order,ORDER_NOT_ON_TICKETputsticket.order_idinexpected. - Never echo upstream tracebacks or raw vendor codes. Map internal exceptions and upstream strings (
E_STOCK_0in Transfer: wrap a new upstream API) into domain error codes.
Repair 4: results sized for the next decision
"Tool definitions and accumulated tool_result blocks consume your context window." (Manage tool context). Every character a read tool returns is billed again on every subsequent model call in the run unless context editing removes it. Anthropic recommends returning high-signal fields, "exposing a simple response_format enum parameter in your tool, allowing your agent to control whether tools return" concise or detailed responses, and adding pagination or explicit truncation limits; as a reference point, "For Claude Code, we restrict tool responses to 25,000 tokens by default." (Writing effective tools for AI agents).
On the six lab tickets, desk_get_ticket_context in concise mode returns 704 to 925 characters (932 mean result characters across entire runs, down from 3,787 in the baseline): the ticket, order line items in integer cents, policy rules, and precomputed checks.within_window so the model never performs calendar math. detailed adds the 5 latest shipping events (stating that 5 of 14 are shown) and prior staff notes. Customer email, phone, street address, fraud_review, and margin_cents never appear (C-10).
{"ticket":{"ticket_id":"T-3031","category":"damaged_on_arrival","status":"open","opened_on":"2026-10-05","customer_message":"The zipper pull on my sleeping bag came off the first time I used it. ..."}, "order":{"order_id":"ORD-7304","delivered_on":"2026-08-20","items":[{"sku":"BAG-20F","name":"20-degree sleeping bag","unit_price_cents":18900,"qty":1}],"total_cents":18900,"refunded_cents":0}, "policy":{"policy_id":"POL-DOA-30","payout_type":"refund","max_payout_cents":6000,"window_days":30,"rule":"Damaged on arrival and reported within 30 days of delivery: ..."}, "checks":{"days_from_delivery_to_report":46,"within_window":false}}Repair 5: side effects in the contract
A contract must declare its side effects honestly, and your handler code must enforce that declaration:
- Reads change nothing (
C-08,C-09). Every contract declareseffect: 'read'oreffect: 'write'. TestC-09invokes every read tool against a clean desk and fails if any state or journal entry changes. - Writes require exact-action approval (
C-14).payouts_issuecallsrequireApprovalwith a fingerprint binding ticket, order, payout type, andamount_centsbefore mutating state (Guide 01, Guide 03). - Writes are idempotent (
C-12). Derivingpayout:<ticket>:<order>:<type>:<amount>lets a repeated call after a network timeout return the original payout withalready_existed: true. - Policy is enforced in handler code (
C-13).payouts_issuere-reads the policy and rejects wrong payout types, expired windows, and over-limit amounts before asking an operator or writing to the ledger.
Mapping side-effect contracts to MCP ToolAnnotations
When you expose these same contracts through an MCP server (Guide 04), the 2026-07-28 MCP schema defines ToolAnnotations ("Optional annotations for the client.", MCP Schema): readOnlyHint (default false), destructiveHint (default true, meaningful when readOnlyHint is false), idempotentHint (default false), and openWorldHint (default true). In our redesigned library, desk_get_ticket_context maps to readOnlyHint: true, openWorldHint: false, while payouts_issue maps to readOnlyHint: false, destructiveHint: false, idempotentHint: true, openWorldHint: false. Treat annotations strictly as UI and routing hints, never as security enforcement: the MCP specification requires clients to "consider tool annotations to be untrusted unless they come from trusted servers." (MCP Tools). Approval, idempotency, and policy checks stay in server code.
LAB-002: redesign the library (40 minutes)
In LAB-002 (hands-on labs), replace src/contracts/redesigned.ts (which starts as a copy of the baseline) across steps R1 to R5 until all three offline gates pass ($0.00):
pnpm lint:contracts --library redesignedpnpm testpnpm eval --library redesignedOn the untouched starter, pnpm test reports # pass 2 and # fail 8; after your redesign it reports # pass 10 and # fail 0, and pnpm eval --library redesigned passes 6 of 6.
Measure the redesign, then decide on discovery
Evaluate contracts against the system of record (OUT-01 to OUT-04) alongside call counts, errors, result characters, and token usage (LAB-003):
pnpm eval --library baseline --model claude --reps 3 --out .runs/eval-baseline.jsonpnpm eval --library redesigned --model claude --reps 3 --out .runs/eval-redesigned.jsonpnpm compare .runs/eval-baseline.json .runs/eval-redesigned.json
Extended description
Across 18 live runs per library (6 synthetic tickets times 3 repetitions on 2026-11-15 with claude-sonnet-5-5 at effort medium), passing runs rose from 6 to 18, mean model calls fell from 7.4 to 3.5, mean uncached input tokens fell from 24,139 to 9,422, and mean tool-result characters fell from 3,787 to 932.
Definition-only overhead (tool_choice: none) rose from +1,096 tokens on the nine terse wrappers to +1,949 tokens on the three strict contracts (including Anthropic's 286-token tool-use system prompt), yet mean estimated run cost dropped from $0.0590 uncached to $0.0244 uncached and $0.0105 with --cache auto (12 runs).
Cost in two views: uncached tokens and automatic prompt caching
Look closely at the definition-only row first (pnpm defs:cost, measured with tool_choice: { type: "none" } on 2026-11-15). On claude-sonnet-5-5, any request carrying tools adds Anthropic's 286-token tool-use system prompt (Pricing). Over a 17-token no-tools prompt, the nine baseline definitions added +1,096 tokens (1,113 total, or 810 pure schema tokens after the 286-token system prompt), while the three redesigned definitions added +1,949 tokens (1,966 total, 1,663 pure schema tokens). Good contracts cost more definition tokens up front. The 39 percent uncached input-token ratio (9,422 versus 24,139) comes from cutting model calls from 7.4 to 3.5, cutting result characters from 3,787 to 932, and eliminating 59 error-probing turns.
Now view the same redesigned library under automatic prompt caching (pnpm eval --cache auto, which sets "Add a single cache_control field at the top level of your request.", Pricing; 512-token minimum on Sonnet 5.5, Prompt caching). At 2026-11-15 Sonnet 5.5 list prices ($2.00 base input, $2.50 5-minute cache write, $0.10 cache hit since 2026-10-07, and $10.00 output per million tokens, Models overview, Pricing), 18 uncached baseline runs cost $1.06 ($0.0590 mean) and 18 uncached redesigned runs cost $0.44 ($0.0244 mean). With --cache auto across 12 runs (6 tickets times 2 repetitions), mean redesigned cost dropped to $0.0105 ($0.0072 on warm second repetitions; 1,701 mean cache-write tokens and 7,707 mean cache-read tokens). Always verify usage fields in your logs: "if both cache_creation_input_tokens and cache_read_input_tokens are 0, the prompt was not cached" (Prompt caching), and editing any tool definition "invalidates the entire cache".
When to add tool discovery
Manage tool context recommends a clear sequence for "Stable tool definitions across many requests": turn on prompt caching first, add tool search at 10 or more tools (or above 10,000 definition tokens, as selection accuracy degrades past 30 to 50 tools), add context editing for long sessions (Guide 08), and reserve programmatic calling for repetitive multi-call loops. Tool search marks long-tail tools defer_loading: true (up to "10,000 tools with defer_loading: true for each request") alongside tool_search_tool_regex_20251119 (case-insensitive Python re.search patterns up to 200 chars) or tool_search_tool_bm25_20251119 (natural-language queries up to 500 chars), and both variants search "tool names, descriptions, argument names, and argument descriptions" (Tool search tool). Two request rules return 400 if broken: "Never set defer_loading: true on the tool search tool itself." (F-11, At least one tool must have defer_loading=false. All tools cannot be deferred.), and "A tool with defer_loading: true can't also carry cache_control: the API returns a 400." (F-10, Tool 'desk_probe_tool_01' cannot have both defer_loading=true and cache_control set. Tools with defer_loading cannot use prompt caching.). Top-level automatic cache_control works cleanly with deferred tools.
To test discovery on 2026-11-15, we added 45 well-formed tools from nine other Example Outfitters departments (inventory_*, shipping_*, vendors_*, hr_*, finance_*, and others) that no support ticket needs (12 live runs per condition, all 12/12 passing):
pnpm eval --library redesigned --discovery catalog-loaded --model claude --reps 2pnpm eval --library redesigned --discovery catalog-deferred --model claude --reps 2pnpm eval --library redesigned --discovery search-all --model claude --reps 2pnpm eval --library redesigned --discovery search-all --search-variant bm25 --model claude --reps 2pnpm defs:costExtended description
Across six tool-loading conditions (18 runs on 3 desk tools; 12 runs on each 48-tool condition, 100% task pass rate), loading all 45 non-desk catalog tools raised definition delta to +8,781 tokens and mean input tokens to 33,332 ($0.0213 cached), while pinning the 3 desk tools and deferring the 45 catalog tools (catalog-deferred) held mean input tokens to 10,766 (+2,334 definition delta, $0.0113 cached, 0 search calls).
Deferring all 48 tools behind regex search cost 30,997 mean input tokens when catalog descriptions ended with Not for customer support tickets, refunds, or store credit., 14,842 tokens with department-only wording (6 domain-noun matches across name, description, argument name, and argument description), and 15,842 tokens under BM25 (25 search calls across 12 runs).
| Condition (12 runs each) | Definition delta | Search calls | Mean uncached input tokens | Uncached mean USD | Cached mean USD (warm rep 2) |
|---|---|---|---|---|---|
3 desk tools only (redesigned, 18 runs) | +1,949 | n/a | 9,422 | $0.0244 | $0.0105 ($0.0072) |
48 tools all loaded (catalog-loaded) | +8,781 | n/a | 33,332 | $0.0722 | $0.0213 ($0.0098) |
3 desk loaded + 45 catalog deferred (catalog-deferred, regex) | +2,334 | 0 | 10,766 | $0.0271 | $0.0113 ($0.0076) |
All 48 deferred, catalog mentions tickets/refunds (search-all, regex) | +671 | 12 | 30,997 | $0.0685 | not run |
All 48 deferred, department-only wording (search-all, regex) | not measured separately | 12 | 14,842 | $0.0362 | not run |
All 48 deferred, catalog mentions tickets/refunds (search-all, BM25) | +643 | 25 | 15,842 | $0.0385 | not run |
Three findings stand out, and the discovery decision holds in both the uncached and prompt-cached views:
- Keep core tools loaded and defer the long tail.
"Keep your 3-5 most frequently used tools non-deferred so Claude can call them without searching first."(Tool search tool; echoed in Introducing advanced tool use,"Keep your three to five most-used tools always loaded, defer the rest."). Incatalog-deferred, Claude made0search calls across12runs, cutting uncached input tokens from33,332to10,766and cached mean cost from$0.0213to$0.0113(-47%, and$0.0076versus$0.0098on warm repetitions). - Regex search matches four fields, not descriptions alone (
F-12). Deferring all48tools looked cheapest indefs:cost(+671tokens) yet averaged30,997input tokens per run when each catalog description ended withNot for customer support tickets, refunds, or store credit.On12of12runs Claude searchedticket|order|policy|refund. Offline field attribution (attributeRegexQueryinsrc/contracts/catalog.ts) shows all45catalog tools matched:41ondescriptionalone and4on multiple fields. Rewriting the catalog to department-only wording (Use it only for inventory work.) dropped mean input tokens to14,842, not all the way to10,766, becauseticket|order|policy|refundstill matched6catalog tools across all four searched fields:inventory_list_low_stock(reorder pointindescription),inventory_get_reorder_plan(reorderinnameandordersindescription),shipping_void_label(refund of postageindescription),vendors_create_purchase_orderandvendors_list_purchase_orders(purchase_orderinnameanddescription), andvendors_record_delivery(purchase_order_idinarg_nameandPurchase order id.inarg_description). - BM25 caps returned tools at 5 by default, and negation boilerplate still hurts ranking. Switching
search-allonmentions-desktotool_search_tool_bm25_20251119(--search-variant bm25; server-side ranking, so offline field attribution applies only to regex) lowered mean input tokens from30,997to15,842($0.0385). Yet on12of12runs, Claude's first query (read support ticketorread ticket) returned five short read-only catalog tools (hr_get_employee,finance_get_tax_summary,vendors_get_vendor,stores_get_store_hours,it_get_device) whose concise descriptions had high term density onread,support, andtickets, missingdesk_get_ticket_contextand forcing a second broader BM25 search (25search calls across12runs;"The response you receive contains server_tool_use blocks showing what ran and what came back.", How tool use works).catalog-deferred(10,766tokens) beatsearch-allunder every search variant.
When to add programmatic tool calling
Programmatic tool calling (code_execution_20260120 or code_execution_20260521 in allowed_callers) lets Claude run loops and filters over tool results inside a code-execution container so intermediate payloads stay out of model context (Programmatic tool calling). Anthropic's documentation reports sharp workload boundaries: "On a 75-tool project-management agent benchmark, enabling programmatic tool calling reduced billed input tokens by roughly 38 percent with no change in task accuracy.", and requests carrying 10 to 49 tools typically save 20 to 40 percent, whereas on τ²-bench (one or two sequential calls per turn) accuracy was unchanged and total cost rose roughly 8 percent: "Sequential single-call workflows do not benefit." (Earlier 2025-11-24 benchmarks in Introducing advanced tool use belong to multi-step research workloads, not 3-call support tickets.) Moreover, "Tools with strict: true are not supported with programmatic calling." Adopting it on Example Outfitters would forfeit strict grammar guarantees on payouts_issue and desk_update_ticket for a 3-step sequential workflow that docs show costs more. We therefore teach it as an explicit "do not adopt here" engineering decision; when a batch fan-out workload does fit (such as ASM-003's 400-payout reconciliation), measure billed input tokens with and without allowed_callers on read-only tools first.
LAB-003: measure and decide (20 minutes)
Follow LAB-003 in the hands-on labs to compare baseline and redesigned offline ($0.00), optionally run the live and --cache auto comparisons, and write a one-page decision record stating your shipping choice, discovery configuration, programmatic-calling decision, and sample limits.
Setup and runtime troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
pnpm preflight fails on Node.js version | Node.js older than 22.18.0, which first ran .ts files without an experimental flag ("v22.18.0 Type stripping no longer emits an experimental warning.", Modules: TypeScript) | Use Node.js 22.23.5 (.node-version), 24.22.3, or Active LTS 26.13.1 |
ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX | Non-erasable TypeScript syntax such as enum or constructor parameter properties | Use string union types and explicit class fields; pnpm typecheck catches this via erasableSyntaxOnly |
400 temperature: Extra inputs are not permitted | Sending temperature, top_p, or top_k on API version 2026-10-21 (@anthropic-ai/sdk 1.x) | Remove sampling parameters and control thoroughness with output_config: { effort: "medium" } |
Error: Could not resolve authentication method... Or for the "Authorization" header to be explicitly omitted | Proxy client using SDK 0.x's defaultHeaders: { 'x-api-key': null } on SDK 1.x | Use defaultHeaders: { Authorization: null }: "If the proxy adds the credential and your client has none, v0 let you say so with either \"X-Api-Key\": null or Authorization: null in defaultHeaders. v1 accepts only Authorization: null." (Migrating to API version 2026-10-21) |
400 when adding input_examples | Example payload violates input_schema (1042 instead of "T-9001") | Validate every example against input_schema and use IDs outside the evaluation set |
Both cache_creation_input_tokens and cache_read_input_tokens stay 0 with --cache auto | Prompt is shorter than Sonnet 5.5's 512-token minimum, or tools changed between turns | Keep tools byte-identical across turns and verify total prompt exceeds 512 tokens |
Transcript looks clean, yet pnpm eval scores safe_stop or wrong_outcome | Tool accepted an ambiguous value (pending) or moved the wrong unit (48 cents) | Inspect pnpm trace <jsonl> alongside OUT-01 to OUT-04 in src/eval/outcome.ts |
| Scripted double behaves differently from Claude | Expected: the double executes five fixed rules in src/model/scripted.ts | Use the double for $0.00 structural regression checks and --model claude for model evidence |
Quick reference
Design rules
- One owner per capability (
C-02). Consolidate reads the agent always chains; split writes across risk classes. - Resource-first names (
C-01).desk_get_ticket_context,payouts_issue,desk_update_ticket. - 3+ sentence descriptions (
C-03). State what it does, when to use it, when not to use it (in its own domain's vocabulary), units, limits, and side effects. - Unambiguous strict inputs (
C-04toC-07). Specific*_idnames with regex patterns, integer*_cents, closedenumsets,explanationinstead of reasoning,strict: true,additionalProperties: false. - Decision-sized results (
C-10). Under2,000characters on read tools,response_format(conciseordetailed) for optional context, zero unneeded contact or margin fields. - Actionable errors (
C-11).is_error: truewithcode,message,field,expected,retryable, andnext_step; never raw stack traces or vendor codes. - Enforced side effects (
C-08,C-09,C-12toC-14). Pure reads; approved, idempotent, policy-checked writes; treat MCPToolAnnotationsas untrusted hints. - Outcome and two-view cost evaluation. Score the system of record (
OUT-01toOUT-04), report both uncached tokens and--cache autodollars, and keep3to5frequent tools loaded when deferring a catalog.
Lab checks
| Check | What it verifies |
|---|---|
| C-01 | Names are namespaced, lowercase, unique (^[a-z][a-z0-9]*(_[a-z0-9]+)+$) |
| C-02 | Each capability has at most one owning tool |
| C-03 | Descriptions have 3+ sentences and a when-not-to-use boundary |
| C-04 | Parameters are described and specifically named (not id, order, ticket, name) |
| C-05 | Money parameters are integer *_cents |
| C-06 | Closed sets declare enum; *_id parameters declare pattern |
| C-07 | strict: true and additionalProperties: false |
| C-08 | effect (read or write) is declared and matches the tool name |
| C-09 | Read tools leave desk state and the write journal untouched |
| C-10 | Read results stay under 2,000 characters and omit contact and margin fields |
| C-11 | Invalid calls return structured JSON errors with code, retryable, and next_step |
| C-12 | Repeating an approved payout with identical inputs pays once (already_existed: true) |
| C-13 | Policy limits, windows, and payout types are enforced in handler code |
| C-14 | Denied operator approval moves zero money and returns retryable: false |
| OUT-01 to OUT-04 | Payouts, final ticket status, matching approvals, and agent-recorded completion verified from desk state |
Commands
pnpm preflight # check Node.js and environment ($0.00)pnpm typecheck # TypeScript 7 erasable-syntax check ($0.00)pnpm lint:contracts --library baseline # static rules C-01 to C-08 ($0.00)pnpm lint:contracts --library redesigned # must print 0 finding(s) ($0.00)pnpm test # C-09 to C-14, outcomes, characterization ($0.00)pnpm self-test # 12 planted checker faults ($0.00)pnpm gallery all # Failure gallery F-01 to F-13 ($0.00)pnpm trace fixtures/transcripts/TR-2-T-3010.jsonl # inspect a recorded transcript ($0.00)pnpm eval --library redesigned # 6-ticket offline evaluation ($0.00)pnpm eval --assessment # 4-task Part A assessment check ($0.00)pnpm eval --library redesigned --model claude --reps 3 --cache auto # optional live cached run (cents)pnpm eval --library redesigned --discovery catalog-deferred --model claude --reps 2pnpm eval --library redesigned --discovery search-all --search-variant bm25 --model claude --reps 2pnpm defs:cost # live definition token cost (cents)pnpm compare .runs/a.json .runs/b.json # side-by-side markdown table ($0.00)Platform limits and version watch list (checked 2026-11-15)
| Item | Verified value on 2026-11-15 | Watch on upgrade |
|---|---|---|
| Strict tool caps (Structured outputs) | 20 strict tools, 24 optional params, 16 union params, 24h grammar cache | Re-probe 20 vs 21 strict tools if compilation limits expand |
tool_choice on claude-sonnet-5-5 (Define tools) | auto and none supported; any and tool return HTTP 400 | Check Define tools model compatibility table on new model releases |
| Tool search (Tool search tool) | Up to 10,000 deferred tools; regex pattern <= 200 chars; BM25 query <= 500 chars; default 5 refs | Keep core tools non-deferred and audit server_tool_use queries |
| Prompt caching & pricing (Prompt caching, Pricing) | 512-token min on Sonnet 5.5; $2.00 in / $2.50 5m write / $0.10 hit / $10.00 out per MTok; 286-token tool system prompt | Update cost constants in scripts/eval.ts when list prices change |
@anthropic-ai/sdk (@anthropic-ai/sdk, Migrating to API version 2026-10-21) | 1.8.0 (Apache-2.0, default API 2026-10-21; temperature/top_p/top_k rejected) | Keep Authorization: null on proxy clients and check role on MessageParam |
| Node.js (Node.js release index and Release WG schedule, Modules: TypeScript) | 22.23.5 pinned (Maintenance LTS, EOL 2027-04-30); verified on 24.22.3 and Active LTS 26.13.1 | Move .node-version to Node.js 26 LTS well before 2027-04-30 |
05Hands-on lab
Lab 2: Redesign the library
Replace src/contracts/redesigned.ts (which starts as a copy of the baseline) with three strict contracts (desk_get_ticket_context, payouts_issue, desk_update_ticket) across repairs R1 to R5 until static lint, all ten handler and outcome tests, and the six-ticket offline evaluation pass at $0.00. All step outputs below were recorded offline on 2026-11-15 on Node.js v22.23.5 and pnpm 12.21.0 at $0.00 (Offline: scripted model double, not Claude. Synthetic data. Simulated services.) inside /tmp/tool-contracts-work; your paths start where you unpacked tool-contracts-labs.zip.
Acceptance checks
- pnpm lint:contracts --library redesigned prints 0 finding(s) across rules C-01 to C-08
- pnpm test passes 10 of 10 checks covering C-09 to C-14 and OUT-01 to OUT-04
- pnpm eval --library redesigned passes 6 of 6 tickets with 0 tool errors and 0 reads that wrote
Lint the redesigned contract file
$ pnpm lint:contracts --library redesignedInitial: Before editing src/contracts/redesigned.ts
$ node scripts/lint.ts --library redesigned Contract lint: redesigned (9 tools) FAIL C-01 Names are namespaced: <resource>_<verb>[_<object>], lowercase, and unique (9) FAIL C-02 Each capability has exactly one owning tool (no overlapping responsibilities) (7) FAIL C-03 Descriptions say what the tool does, when to use it, and when not to (3+ sentences) (17) FAIL C-04 Every parameter is described and specifically named (no bare id, query, amount, type, name, fields) (32) FAIL C-05 Money is integer cents in a parameter whose name ends in _cents (4) FAIL C-06 Closed sets use enum; *_id parameters carry a pattern (3) FAIL C-07 Strict schemas: strict true, additionalProperties false, every property listed (18) FAIL C-08 Effects are declared: read tools use read verbs, write tools say what they change (4) Findings:Final: After repairs R1 to R5
$ node scripts/lint.ts --library redesigned Contract lint: redesigned (3 tools) PASS C-01 Names are namespaced: <resource>_<verb>[_<object>], lowercase, and unique PASS C-02 Each capability has exactly one owning tool (no overlapping responsibilities) PASS C-03 Descriptions say what the tool does, when to use it, and when not to (3+ sentences) PASS C-04 Every parameter is described and specifically named (no bare id, query, amount, type, name, fields) PASS C-05 Money is integer cents in a parameter whose name ends in _cents PASS C-06 Closed sets use enum; *_id parameters carry a pattern PASS C-07 Strict schemas: strict true, additionalProperties false, every property listed PASS C-08 Effects are declared: read tools use read verbs, write tools say what they change 0 finding(s).Run the handler behavior and outcome test suite
$ pnpm testInitial: Before editing src/contracts/redesigned.ts
ok 1 - baseline fails every lint rule ok 2 - baseline reproduces the unit trap and the wrong-tool substitution with the double not ok 3 - C-01 to C-08: the contract lint is clean not ok 4 - C-09: read tools change nothing in the system of record not ok 5 - C-10: read results stay under budget and carry no customer contact or internal data not ok 6 - C-11: every tool returns structured, actionable errors for an unknown ticket not ok 7 - C-12: repeating the same payout call pays once not ok 8 - C-13: the policy limit is enforced in code, with a next step that hands off not ok 9 - C-14: a denied approval moves no money not ok 10 - OUT-01 to OUT-04: all six tasks pass with the redesigned library # tests 10 # pass 2 # fail 8Final: After repairs R1 to R5
ok 1 - baseline fails every lint rule ok 2 - baseline reproduces the unit trap and the wrong-tool substitution with the double ok 3 - C-01 to C-08: the contract lint is clean ok 4 - C-09: read tools change nothing in the system of record ok 5 - C-10: read results stay under budget and carry no customer contact or internal data ok 6 - C-11: every tool returns structured, actionable errors for an unknown ticket ok 7 - C-12: repeating the same payout call pays once ok 8 - C-13: the policy limit is enforced in code, with a next step that hands off ok 9 - C-14: a denied approval moves no money ok 10 - OUT-01 to OUT-04: all six tasks pass with the redesigned library # tests 10 # pass 10 # fail 0Evaluate all six support tickets against the desk record
$ pnpm eval --library redesigned$ node scripts/eval.ts --library redesigned Offline: scripted model double, not Claude. Synthetic data. Simulated services. Condition redesigned:scripted: 3 tool definitions (4119 characters) ticket rep verdict calls tools errors in_tok result_chars stop T-1042 1 pass 4 3 0 0 1083 end_turn T-2077 1 pass 4 3 0 0 998 end_turn T-3010 1 pass 3 2 0 0 749 end_turn T-3022 1 pass 3 2 0 0 867 end_turn T-3031 1 pass 3 2 0 0 895 end_turn T-3045 1 pass 4 3 0 0 1002 end_turn Summary: 6/6 pass, 0 safe stop, 0 wrong outcome; mean 3.5 model calls, 2.5 tool calls; 0 tool errors; mean 932 result chars per runReproduce the baseline pending status trap
$ pnpm gallery f-02$ node scripts/gallery.ts f-02 F-02 [Reproducible] update_ticket rejects "escalated" and maps "pending" silently back to "open" Evidence: status="escalated" -> Error: invalid value; status="pending" -> {"status":"ok"}, stored ticket.status="open"Verify all 12 planted checker faults
$ pnpm self-test$ node scripts/self-test.ts PASS FLT-00: clean fixture passes C-01..C-08 with 0 findings PASS FLT-C-01: planted fault triggers C-01 independently PASS FLT-C-02: planted fault triggers C-02 independently PASS FLT-C-03: planted fault triggers C-03 independently PASS FLT-C-04: planted fault triggers C-04 independently PASS FLT-C-05: planted fault triggers C-05 independently PASS FLT-C-06: planted fault triggers C-06 independently PASS FLT-C-07: planted fault triggers C-07 independently PASS FLT-C-08: planted fault triggers C-08 independently PASS FLT-OUT: checkOutcome catches planted payout, shipment, status, approval, and read-write faults PASS FLT-STEP-0: runAgent returns steps=0 when model.create throws on call 1 PASS FLT-ATTR: attributeRegexQuery rejects empty input and separates name, description, arg_name, arg_description self-test: PASS (12 checks verified)Show the solution discussion
Inspect the step-by-step checkpoints R1 through R5 or copy the finished reference into a separate scratch folder with node ../solutions/apply-solution.mjs final /tmp/contracts-final so you do not overwrite your own work.
desk_get_ticket_context folds ticket, order, policy and precomputed checks.within_window into one pure read; payouts_issue enforces policy windows and caps, derives a deterministic idempotency key and calls requireApproval before mutating state; desk_update_ticket restricts status to resolved and needs_human and rejects RESOLVE_WITHOUT_ACTION.
More labs in this guide
Lab 1: Diagnose the baseline from transcripts
Run static contract lint and the offline evaluation on the nine-tool baseline, then trace three unedited 2026-11-15 Claude transcripts (TR-1-T-2077, TR-2-T-3010, TR-3-T-1042) to map every failure to D-1 through D-5 or read-that-writes with its step number, 12-character call id, and contract fix.
Lab 3: Measure and decide
Compare baseline and redesigned side by side offline ($0.00), optionally run a bounded 1-repetition live and --cache auto comparison on claude-sonnet-5-5, inspect regex field attribution with pnpm gallery f-12, and write a one-page engineering decision record.
06Failure gallery
Thirteen ways this breaks, and how you will know
Eight of these reproduce in the lab; the other five were observed in this guide's recorded runs. Learn the symptom first; it is what you will actually see.
- F-01
gallery f-01ReproducibleRead tool mutates ticket assignment
- Symptom
- Opening a ticket with get_ticket reassigns ticket.assignee to "agent" and updates last_viewed_at, recording a write on all 18 of 18 baseline runs.
- Diagnosis
- Thin wrappers around legacy helpdesk endpoints inherit implicit "claim on view" mutations, turning a harmless read retry into a queue state change.
- Fix
- Remove ctx.desk.mutate from read handlers and declare effect: 'read' on desk_get_ticket_context.
- Verify
- "pnpm gallery f-01" shows the baseline journal entry; test C-09 in pnpm test confirms zero mutations across all redesigned reads.
- F-02
gallery f-02ReproducibleThe pending status trap
- Symptom
- update_ticket rejects escalated, handed_to_human, and pending_human with "Error: invalid value", then returns {"status":"ok"} for status: "pending" while leaving ticket.status set to "open" (11 of 12 live safe stops).
- Diagnosis
- status is typed as an unconstrained string backed by a hidden legacy map where pending translates to open.
- Fix
- Define status: { type: "string", enum: ["resolved", "needs_human"] } with strict: true on desk_update_ticket, and verify outcomes from the desk record rather than model prose.
- Verify
- "pnpm gallery f-02" reproduces the "ok" response and "open" stored status; C-06, C-07, and OUT-02 verify the enum and outcome check.
- F-03
gallery f-03ReproducibleDollar-versus-cent unit mismatch pays 48 cents
- Symptom
- On T-1042 ($48.00 tent pole), the scripted double calls issue_refund with amount: 48, paying $0.48 (48 cents) instead of $48.00 and failing OUT-01.
- Diagnosis
- issue_refund names its parameter amount: number with no unit in the name or description while add_credit takes dollars: number.
- Fix
- Name every currency field *_cents (amount_cents), set type: "integer", and include "4800 means $48.00" in the parameter description.
- Verify
- "pnpm gallery f-03" reproduces verdict=wrong_outcome on T-1042; C-05 and "pnpm eval --library redesigned" confirm 4800 cents.
- F-04
gallery f-04Reproducibleclose_ticket resolves an escalation after update_ticket fails
- Symptom
- On over-limit ticket T-3022, after update_ticket rejects escalated, the scripted double switches to close_ticket and marks the escalation resolved with zero payout (wrong_outcome).
- Diagnosis
- Two tools (update_ticket and close_ticket) share ticket.status, and close_ticket sets resolved unconditionally without checking whether the policy's action occurred.
- Fix
- Give ticket.status a single owner (desk_update_ticket) and return RESOLVE_WITHOUT_ACTION (expected: "needs_human") when resolved is requested with no recorded payout or replacement.
- Verify
- "pnpm gallery f-04" reproduces the false resolution; C-02 and pnpm test verify single ownership and RESOLVE_WITHOUT_ACTION.
- F-05
gallery f-05ReproducibleOverlapping refund wrapper bypasses approval and idempotency
- Symptom
- Calling issue_refund twice under a denying approver still records two unapproved $48.00 refunds (9600 cents total, approved: false, idempotency_key: null).
- Diagnosis
- Legacy issue_refund remained in the tool list alongside create_refund_v2, leaving an unguarded write path one tool-selection mistake away.
- Fix
- Retire issue_refund and add_credit; consolidate money movement into payouts_issue gated by requireApproval and deterministic key payout:<ticket>:<order>:<type>:<amount>.
- Verify
- "pnpm gallery f-05" reproduces the double unapproved write; C-12 and C-14 prove idempotency and denied-approval safety.
- F-06
gallery f-06ReproducibleHidden policy IDs and raw stack traces
- Symptom
- get_policy {"name": "damaged_on_arrival"} returns {"error":"NOT_FOUND"} (21 chars), and get_order on an unknown ID returns "TypeError: Cannot read properties of undefined (reading 'lines')" with internal server file paths.
- Diagnosis
- get_policy expects POL-DOA-30 while asking for name, and get_order lets an uncaught exception serialize into tool_result.
- Fix
- Fold policy lookup by ticket.category into desk_get_ticket_context, and return structured fail() objects (is_error: true, code, field, expected, retryable, next_step) on every error path.
- Verify
- "pnpm gallery f-06" shows both raw baseline errors; C-11 tests invalid inputs across every redesigned contract.
- F-07
gallery f-07ReproducibleEndpoint dump leaks contact data and internal margins
- Symptom
- get_order on ORD-5521 returns nearly 3,000 characters including @example.com email, +1-555-0110 phone, street address, 14 shipping events, fraud_review, and margin_cents.
- Diagnosis
- Returning the upstream ORM object exposes personal and internal fields no support rule uses and inflates every subsequent turn's prompt tokens.
- Fix
- Project only decision fields in desk_get_ticket_context (704 to 925 chars in concise mode), gate shipping history behind response_format: "detailed", and omit customer.read entirely.
- Verify
- "pnpm gallery f-07" flags the size and leaked fields; C-10 enforces < 2,000 characters and zero contact or margin keys across all tasks.
- F-08Observed
Forced tool_choice (any or tool) returns HTTP 400 on Sonnet 5.5
- Symptom
- Sending tool_choice: { type: "any" } or tool_choice: { type: "tool", name: "..." } to claude-sonnet-5-5 fails immediately with HTTP 400 invalid_request_error: tool_choice: type "tool" and "any" are not supported for this model.
- Diagnosis
- Claude Sonnet 5.5, Opus 5.5, Fable 5.2 and 5.1, and Mythos 5.2 and 5.1 reject forced tool selection (Claude API docs, Define tools).
- Fix
- Keep tool_choice at auto with strict: true tool schemas and clear system instructions, or use output_config.format structured outputs when you need a single JSON response rather than a tool loop.
- Verify
- Run "pnpm gallery f-08" to inspect the recorded 2026-11-15 error, and verify src/model/anthropic.ts sends tool_choice only for defs:cost (none).
- F-09Observed
21st strict tool or 25th optional parameter fails grammar compilation
- Symptom
- Marking a 21st tool strict: true returns HTTP 400: "Too many strict tools (21). The maximum number of strict tools supported is 20. Try reducing the number of tools marked as strict." Adding a 25th optional parameter across strict schemas returns HTTP 400: "Schemas contains too many optional parameters (25), which would make grammar compilation inefficient. Reduce the number of optional parameters in your tool schemas (limit: 24)."
- Diagnosis
- Grammar compilation enforces per-request caps of 20 strict tools, 24 optional parameters, and 16 union parameters (Claude API docs, Structured outputs).
- Fix
- Reserve strict: true for write tools and high-risk schemas, keep required parameters required unless genuinely optional, and leave low-risk read catalog tools non-strict.
- Verify
- "pnpm gallery f-09" prints both recorded 2026-11-15 400 messages; catalog.ts keeps the 45 non-desk catalog tools non-strict so 48-tool runs stay at 3 strict tools.
- F-10Observed
cache_control on a defer_loading: true tool returns HTTP 400
- Symptom
- Attaching cache_control: { type: "ephemeral" } directly to a tool definition that also sets defer_loading: true returns HTTP 400: "Tool 'desk_probe_tool_01' cannot have both defer_loading=true and cache_control set. Tools with defer_loading cannot use prompt caching."
- Diagnosis
- Deferred definitions are withheld from the initial prompt prefix, so block-level cache_control on a deferred tool entry is invalid (Claude API docs, Tool search tool).
- Fix
- Remove cache_control from deferred tool entries and enable automatic prompt caching with a single top-level cache_control: { type: "ephemeral" } field on the messages.create request (--cache auto).
- Verify
- "pnpm gallery f-10" prints the recorded 400 error; "pnpm eval --library redesigned --discovery catalog-deferred --cache auto" succeeds with cache writes and hits.
- F-11Observed
Deferring every tool including the search tool returns HTTP 400
- Symptom
- Setting defer_loading: true on every entry in tools returns HTTP 400: "At least one tool must have defer_loading=false. All tools cannot be deferred."
- Diagnosis
- When every tool is deferred, Claude has no loaded tool (not even the search tool) to discover the rest (Claude API docs, Tool search tool).
- Fix
- Never set defer_loading: true on tool_search_tool_regex_20251119 or tool_search_tool_bm25_20251119, and keep your 3 to 5 core workflow tools non-deferred.
- Verify
- "pnpm gallery f-11" prints the recorded 400 error; src/model/anthropic.ts leaves the search tool non-deferred in both catalog-deferred and search-all.
- F-12
gallery f-12ReproducibleNegation boilerplate makes search-all load unrelated tools
- Symptom
- Putting "Not for customer support tickets, refunds, or store credit." in 45 non-desk catalog tools caused regex search-all (ticket|order|policy|refund) to match 45/45 catalog tools (30,997 mean input tokens), and caused BM25 search-all ("read support ticket") to miss desk_get_ticket_context on 12 of 12 initial queries (25 search calls total).
- Diagnosis
- Both regex and BM25 search tool names, descriptions, argument names, and argument descriptions (Claude API docs, Tool search tool). Naming another domain in a "when not to use" clause plants that domain's keywords into every deferred tool, and domain nouns like reorder or purchase_order_id still match order across 6 department-only tools.
- Fix
- Keep core tools non-deferred (catalog-deferred, 10,766 tokens) and write deferred descriptions in the positive vocabulary of their own domain ("Use it only for inventory work.").
- Verify
- "pnpm gallery f-12" runs attributeRegexQuery across both catalog variants and prints the 45/45 versus 6/45 field breakdown.
- F-13Observed
Beta tool_addition naming an undeclared tool returns tool_reference_unresolved
- Symptom
- Under anthropic-beta: inline-tools-2026-09-15, sending a mid-conversation role: "system" message with tool_addition referencing undeclared_desk_tool returns HTTP 400 (tool_reference_unresolved): "messages.1.content.0: tool_addition/tool_removal references unknown tool 'undeclared_desk_tool'". Placing that role: "system" message directly after an assistant text turn also returns 400 unless it follows a user message or a server-tool result.
- Diagnosis
- By-reference tool_addition can only surface tools already declared in the top-level tools array (typically with defer_loading: true), and system messages carry strict turn-order placement rules (Claude API docs, Mid-conversation system messages and tool changes).
- Fix
- Declare every candidate tool in tools up front with defer_loading: true, and inject role: "system" tool_addition blocks immediately after a user message.
- Verify
- "pnpm gallery f-13" prints the recorded 2026-11-15 tool_reference_unresolved error.
07Assessment
Transfer task and rubric
Wrap the simulated Example Parcel carrier API in one strict replacement-shipment write contract, diagnose an unedited baseline Claude transcript, and justify discovery and calling choices.
| Criterion | 0 · Emerging | 1 · Developing | 2 · Proficient | 3 · Exemplary |
|---|---|---|---|---|
| R-A1 Responsibility and name (OBJ-002) | No replacement tool added, or the tool claims an existing capability | Tool runs, but its name is verb-first or its description omits when-not-to-use or side-effect boundaries | Resource-first lowercase name, single capability shipment.replacement, and a 3+ sentence description stating when to use, when not to use, and what record it changes | Proficient, and rationale.md explains why replacement belongs in its own write tool rather than a third payout_type on payouts_issue |
| R-A2 Inputs and least data (OBJ-003) | Model supplies a street address, customer contact field, or unvalidated free-form object | Specific parameter names, but pattern, strict: true, or additionalProperties: false is missing | ticket_id and sku (or equivalent minimal identifiers) with regex patterns, strict: true, additionalProperties: false, and zero contact fields | Proficient, and the handler rejects a SKU that is not on the ticket's order with the ordered SKU named in expected |
| R-A3 Actionable errors (OBJ-004) | Raw carrier codes (E_STOCK_0) or stack traces reach the model | Some carrier codes are mapped, but retryable, field, or next_step is missing or misleading | Out-of-stock, wrong-SKU, wrong-category, and non-automated-policy cases each return a stable code, honest retryable, and next_step | Proficient, and E_DUP_REF is handled as an idempotent already_existed: true result with rationale.md explaining why a duplicate reference is not an agent error |
| R-A4 Decision-sized results (OBJ-004) | Returns the raw carrier payload | Trimmed payload that still leaks label_zpl_base64, rates_considered, or address fields | Returns shipment_id, sku, qty, eta_business_days, and already_existed in under 600 characters with zero contact or internal fields | Proficient, and rationale.md names which returned fields the agent needs to write the resolution note on desk_update_ticket |
| R-A5 Enforced side effects (OBJ-005) | Ships twice on a retry, skips operator approval, or ships on a damaged-item ticket | Idempotent and approved, but policy (payout_type, automated) is stated only in prose rather than enforced in handler code | Declared write, gated by requireApproval, idempotent via a deterministic carrier ref, and checked against payout_type and automated in code | Proficient, and the submission verifies that T-3010 still escalates on the main seed where POL-WRONG-30 has automated: false |
| R-A6 Interface rationale (OBJ-002 to OBJ-005) | Missing or under 80 words | Summarizes the code line by line without explaining design tradeoffs | 150 to 300 words covering model inputs, handler lookups, result trimming, error mapping, idempotency, and one rejected design | Proficient, and the rejected design is a plausible engineering alternative analyzed with its concrete token, safety, or schema cost |
| R-B1 Transcript diagnosis (OBJ-001) | Restates the transcript narrative without defect classes | Names defect classes without matching step numbers and 12-character call ids | Identifies at least three distinct defect classes in ASM-002-T-3031.jsonl with exact step and call id, and explains step 6's pending to open mapping and harness safe stop | Proficient, and ranks the top two contract repairs by exact steps saved on T-3031, comparing baseline's 7 steps with the 3-step redesigned pattern |
| R-C1 Architecture decision (OBJ-006) | Gives opinions with neither measured numbers nor documented platform rules | Cites either measured numbers or platform rules, but not both, or omits sample limits | Both answers cite at least one 2026-11-15 measurement from "Measure the redesign, then decide on discovery" and one documented platform constraint, and state one limit of the evidence | Proficient, and names the concrete pre-release experiment (including BM25 versus regex query inspection or allowed_callers read-only scoping) that could overturn the choice |
Proficiency standard: Proficient or higher on every criterion. Proposed local standard, not an official certification.
The full brief
In Part A (ASM-001), Example Outfitters enables automated replacements for wrong-item tickets on the assessment seed, backed by the simulated Example Parcel endpoint in src/services/carrier.ts. You add one declared write tool (shipment.replacement) to src/contracts/redesigned.ts without breaking the three existing contracts, keep customer contact data and base64 labels out of model inputs and results, map upstream carrier failures (E_STOCK_0, E_ADDR_17, E_DUP_REF, E_SVC) to actionable structured errors or idempotent outcomes, and submit a 150 to 300 word interface rationale.
In Part B (ASM-002), you trace fixtures/transcripts/ASM-002-T-3031.jsonl (Live: claude-sonnet-5-5, effort medium. Recorded 2026-11-15) with pnpm trace, classify every observed contract defect by step number and tool_use call id, explain why step 6 returned {"status":"ok"} while the run still ended in a harness safe stop, and rank the two contract changes that remove the most wasted steps.
In Part C (ASM-003), you write two 4 to 6 sentence engineering decisions from the guide's 2026-11-15 numbers and documented platform constraints: how to configure tool loading and search for a 120-tool multi-department support agent, and whether to adopt programmatic tool calling for a 400-payout nightly reconciliation job while keeping money-moving tools protected.
Passing requires Proficient (2) or higher on every criterion (R-A1 to R-A6, R-B1, R-C1) and no critical failure (CF-1 a write without operator approval or an idempotency key; CF-2 money accepted or returned as floating-point dollars; CF-3 a read tool that mutates desk state; CF-4 a tool input or result that exposes customer street address, email, phone, fraud score, or margin data). On 2026-11-15, the reference assessment solution passed 4 of 4 live tasks (3.8 mean model calls, 11,561 mean input tokens, $0.0284 mean run cost).
08Instructor kit
Teach it to a room
Facilitation guide and lab solutions
A 125-minute workshop plan across LAB-001 to LAB-003 and the three-part transfer assessment, with hint ladders, live-room troubleshooting, and a lab solutions guide whose offline verification script checks every checkpoint and negative break test at $0.00.
Delivery for 12-person and 100-person rooms
Concrete facilitation plans for an interactive 12-person cohort (with optional capped live comparisons) and a 100-person room running 100 percent offline with table leads and timed checkpoints. Newly authored planning designs; no cohort has run them yet.
Dry run and observation rubric
A proposed 60-minute observed teach-back, live contract-repair drill, transcript diagnosis walk-through, and blind calibration scoring against four reference submissions. No dry run has been held yet.
Clearance record
A per-instructor, per-version clearance template filled in only from an observed dry run and signed by an authorized reviewer. Blank: nobody has been cleared to teach this guide.
Evaluation plan and pass-rate template
Pre-workshop diagnostic, lab checkpoint tracking, double-scored calibration protocol, learner privacy rules, and a cohort pass-rate report template with numerators and denominators. Blank until piloted.
10Sources and verification
- [1]Anthropic Engineering, Writing effective tools for AI agents, using AI agents (published 2025-09-11) · anthropic.comchecked Nov 15, 2026
- [2]Claude API docs, Define tools · platform.claude.comchecked Nov 15, 2026
- [3]Claude API docs, Strict tool use · platform.claude.comchecked Nov 15, 2026
- [4]Claude API docs, Structured outputs · platform.claude.comchecked Nov 15, 2026
- [5]Claude API docs, Handle tool calls · platform.claude.comchecked Nov 15, 2026
- [6]Claude API docs, How tool use works · platform.claude.comchecked Nov 15, 2026
- [7]Claude API docs, Tool search tool · platform.claude.comchecked Nov 15, 2026
- [8]Claude API docs, Programmatic tool calling · platform.claude.comchecked Nov 15, 2026
- [9]Claude API docs, Models overview · platform.claude.comchecked Nov 15, 2026
- [10]Claude API docs, Effort · platform.claude.comchecked Nov 15, 2026
- [11]npm registry, @anthropic-ai/sdk · npmjs.comchecked Nov 15, 2026
- [12]Node.js release index and Release WG schedule · nodejs.orgchecked Nov 15, 2026
- [13]Node.js 22 documentation, Modules: TypeScript · nodejs.orgchecked Nov 15, 2026
- [14]npm registry, pnpm, typescript, and @types/node · npmjs.comchecked Nov 15, 2026
- [15]Claude API docs, Manage tool context · platform.claude.comchecked Nov 15, 2026
- [16]Claude API docs, Prompt caching · platform.claude.comchecked Nov 15, 2026
- [17]Claude API docs, Mid-conversation system messages and tool changes · platform.claude.comchecked Nov 15, 2026
- [18]Claude API docs, Pricing · platform.claude.comchecked Nov 15, 2026
- [19]Claude API docs, Migrating to API version 2026-10-21 · platform.claude.comchecked Nov 15, 2026
- [20]Claude API docs, Release notes · platform.claude.comchecked Nov 15, 2026
- [21]Anthropic Engineering, Introducing advanced tool use on the Claude Developer Platform (published 2025-11-24) · anthropic.comchecked Nov 15, 2026
- [22]Model Context Protocol specification 2026-07-28, Server features: Tools · modelcontextprotocol.iochecked Nov 15, 2026
- [23]Model Context Protocol specification 2026-07-28, Schema reference (ToolAnnotations) · modelcontextprotocol.iochecked Nov 15, 2026
- [24]University of Hawai'i at Manoa Assessment and Curriculum Support Center, Creating and Using Rubrics · manoa.hawaii.educhecked Nov 15, 2026
- [25]Carnegie Mellon University Eberly Center, Creating and Using Rubrics · cmu.educhecked Nov 15, 2026
- [26]X Help Center, How to Post (and LinkedIn Help post limit check) · help.x.comchecked Nov 15, 2026
Independent work. Not affiliated with or endorsed by any vendor named in this guide.
