Context Engineering for Long-Running Agents

You need two ideas from the earlier guides, and the download carries the code for both.

The loop. Guide 01's agent runs a bounded tool-use loop for the Example Outfitters support desk. Claude answers a request with text, a tool call or both; the loop runs each tool, returns the result as a tool_result block in the next user message, and repeats until Claude stops or a limit is hit. Each turn of that loop is one Messages API request, and each request carries the whole conversation so far. That last fact is why this guide exists: in a long conversation, every tool result the agent ever saw rides along in every later request unless something removes it.

The grade. Guide 02 grades the outcome, not the transcript. Its state grader reads the desk's records (the ticket's status, the payouts written, the internal notes) and compares them with what the task requires; a reply that claims a refund proves nothing until the payout exists. Its claim grader checks each claim in a reply against a digest of those records. Results are reported as counts over repeated runs, failures included.

This guide's kit is Guide 02's published lab download, byte for byte, apart from the SDK version in package.json and its lockfile, with the context kit added beside it. GUIDE-02-FIXTURE.md in the download names the archive and its SHA-256, and CONTEXT-KIT.md lists what was added.

What a context window holds

Anthropic's context windows page defines the window as "all the text a language model can reference when generating a response, including the response itself." Everything you send counts toward it: the system prompt, every message including tool results, and the tool definitions. The same page warns that "As token count grows, accuracy and recall degrade, a phenomenon known as context rot", which is why a larger window is not a reason to fill it. Anthropic's engineering article on context engineering calls context "a finite resource with diminishing marginal returns" (Anthropic, 2025).

Two research results stand behind that warning, and both are worth reading with their limits in mind. Liu and colleagues found that models used information at the start or end of a long input more reliably than information in the middle (Liu et al., first posted in 2023 and published, peer reviewed, in TACL). Chroma's "Context Rot" report measured 18 models and found that performance "varies significantly as input length changes, even on simple tasks" (Hong, Troynikov and Huber, 2025); it is a technical report without peer review, from a company that sells a vector database. Neither measured this guide's shift, and this guide does not test whether its strategies avoid or cause context rot. It measures task outcomes.

Four kinds of context, and history

Every block in a request does one of five jobs. The inventory in Reading a context inventory tags each block with one of them.

The distinction that matters most is between evidence and its source. An order record in a tool result is a copy, true when it was read. The order itself lives in the desk's systems of record, and it can change after the copy was made. The shift seeds exactly that case.

Four moves

A strategy is a set of rules for four moves:

• Load: put something in the window up front, before any ticket asks for it.
• Retain: keep something in the window from one request to the next.
• Summarize: replace part of the conversation with a shorter account of it.
• Retrieve: fetch something into the window when a ticket needs it, by an identifier such as an order id or a policy id.

The article describes the retrieve move as agents that "maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools" (Anthropic, 2025). The Claude API supplies the mechanisms for the other moves: context editing removes old tool results from what Claude sees, the memory tool gives the agent files outside the window, and compaction replaces the conversation with a summary block. Three strategies shows each one as the kit uses it.

What stays outside the window

Two subjects sit next to context selection and are often confused with it.

• Long-term application storage. The desk's tickets, orders, payouts and policies are the systems of record. The memory directory is not one of them: a note that records a $30.00 refund is context the agent wrote, and only the payouts table says whether the refund exists. The checks in this guide read the systems of record, never the notes.
• Durable execution. Resuming an agent safely after a crash, without repeating a refund it already issued, is a separate subject with its own controls. This guide's shift runs in one process from the first ticket to the last.

The long shift

The kit's generator writes one synthetic shift from a seed: pnpm shift:generate with seed 8 writes data/shift/, whose hash is in data/shift/SHIFT-HASH.txt. The shift is Monday, 2026-11-02, from 09:00 to 11:45 UTC: 12 tickets arriving every 15 minutes, worked in one conversation. Each ticket arrives as a new user message, and the agent finishes it before the next one arrives.

The queue and its traps

The tickets cover three policy families: damaged on arrival (POL-DOA-30), late delivery (POL-LATE-14) and wrong item (POL-WRONG-30). Three traps make context matter.

• A policy that changes mid-shift. At 10:25 UTC, POL-DOA-30 version 5 replaces version 4 and the damage refund limit drops from $60.00 to $40.00. Tickets that arrive after 10:25 must follow version 5.
• An order that changes after the agent reads it. At 09:45 the agent reads order ORD-3104 for ticket T-2104. At 10:05, finance refunds that order in full, so its refundable balance becomes $0.00. At 11:00, ticket T-2109 asks for a damage refund on the same order. A copy of the order read at 09:45 says there is money left; the system of record says there is none.
• A standing instruction that must last the shift. With ticket T-2102 at 09:15, the operations lead puts order ORD-3111 on hold during a chargeback review: no refund or credit on it, whatever the policy allows, and any ticket about it goes to a human with a note that names the hold. The instruction arrives as a role: "system" message in the conversation. Ticket T-2111 asks for a refund on ORD-3111 at 11:30, more than two hours and many requests later.

Three more tickets test what the earlier guides tested: T-2110 asks again for a refund T-2105 already paid, and two tickets need a human whatever the context holds, T-2102 because only a person can arrange a replacement for a wrong item and T-2104 because only a person can issue store credit for a late delivery.

The checks

The shift's checks sit beside Guide 02's graders, in evals/shift-checks.ts, and run on the desk's records after a run. Guide 02's own graders are unchanged.

A ticket passes when every check that applies to it passes. Not every check applies to every ticket: C-HOLD applies only to T-2111, C-FRESH only to T-2109, and C-POLICY to the 10 damaged-on-arrival tickets, whose policy lets the agent refund. The results in Measuring the strategies report each check over the tickets it applies to.

The claim grader is a model, and a model grader can be wrong. Guide 02's lesson applies: calibrate it on cases with known answers and read every failure it reports. Measuring the strategies says how the record digest it reads was corrected during this guide's measurements, and why.

Three strategies

Each strategy is a JSON file in strategies/, named by its id and version, such as S1.v1.json. The runner hashes the file and writes the hash into every run record, so a result always names the exact strategy that produced it. A strategy sets what is loaded up front, whether operator notices about changes arrive, which tools the agent has, and the clearing, memory and compaction settings. The model and effort are the same for every strategy: claude-sonnet-5-5 at effort medium, Guide 02's defaults.

Every API behavior below was read in Anthropic's documentation and tested on the pinned SDK, @anthropic-ai/sdk 1.3.0, on 2026-10-29. Context editing and compaction are in beta, so recheck them before you rely on them.

S0: load everything

S0 loads the whole library into the system prompt before the first ticket: every policy version, every order, 80 earlier closed tickets and eight reference documents. Each ticket's text arrives in its user turn. The agent has two tools, issue_refund and update_ticket, and none to fetch an order or a policy, because everything is already there. Operator notices in the conversation tell it when the policy changes and when finance refunds ORD-3104. Nothing is cleared and nothing is summarized.

The first request of every S0 run carried 40,291 input tokens by the usage the API returned, nearly all of it the system prompt. That sounds expensive, and per request it is large, but prompt caching changes the price. The kit puts a cache breakpoint at the end of the system prompt and turns on automatic caching with a top-level cache_control, which "caches all content up to and including the last cacheable block", so "the cache point moves forward automatically as conversations grow". In the three-ticket spike run on 2026-10-29, the first request wrote 40,290 tokens to the cache and read none; every later request read 40,290 to 42,558 tokens from it and wrote 352 to 431. On Sonnet 5.5, a cache hit costs $0.10 per million tokens against $2 for uncached input (pricing, read 2026-10-29). The cache lasts five minutes by default, and "The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response" (prompt caching), so a request that starts more than five minutes after the one before it finds the cache cold. In the measured runs, no two requests were more than 15.3 seconds apart, and each run's first request was its only cold one, because each run's system prompt opens with its run id. The run records mark each request's cache state.

S0 has two weaknesses that caching does not touch. Every superseded policy version, and the 09:00 copy of every order, sits in every request after it stops being current, so the agent must notice which version and which status hold now. And the window keeps growing, ticket after ticket.

S1: retrieve on demand, clear old results

S1 starts with a short system prompt: the desk's instructions and the identifiers the agent needs, with no policies or orders. Its first request carried 1,564 input tokens, the tool definitions included. Each user turn names the next ticket by id, and three tools fetch what it needs: get_ticket, get_order, which returns the order as it stands at the moment of the call, and search_policy, which returns the version in force at that moment. No notices arrive; the agent learns about changes by fetching, and its instructions say to read the order and the policy again for every ticket rather than rely on an earlier result.

Fetching keeps each copy fresh, but the copies pile up as old tool results. S1 removes them with context editing. The request carries the beta header context-management-2025-06-27 and a context_management field:

{  "edits": [    {      "type": "clear_tool_uses_20250919",      "trigger": { "type": "input_tokens", "value": 12000 },      "keep": { "type": "tool_uses", "value": 3 },      "clear_at_least": { "type": "input_tokens", "value": 4000 },      "clear_tool_inputs": false    }  ]}

When a request's input passes the trigger, "the API automatically clears the oldest tool results in chronological order", and it "replaces each cleared result with placeholder text indicating to Claude that it was removed" (context editing). It keeps the most recent tool uses, the keep value, and with clear_tool_inputs left at false the tool calls themselves stay, so Claude can still see what it asked for. The clearing happens on the server: "Your client application maintains the full, unmodified conversation history", so the runner's own copy of the conversation stays whole, and each response reports what was cleared in context_management.applied_edits.

The documentation lists defaults and leaves the minimums unstated, so the spike tried the edges on 2026-10-29. A trigger of 0 is refused ("Input should be greater than or equal to 1"). A tool_uses trigger of 1 with the default keep of 3 is refused, because keep must not exceed the trigger. A clear_at_least larger than what can be cleared means nothing is cleared. The token counting endpoint previews a clear: it returns the count after clearing and context_management.original_input_tokens before it.

Clearing has a price that the documentation states plainly: it "Invalidates cached prompt prefixes when content is cleared", and "You'll incur cache write costs each time content is cleared" (context editing). A change in what is cleared changes the prompt from the first cleared result onward, and a prompt the cache does not hold is written to it again from there. clear_at_least exists for that: the documentation's advice is to "clear enough tokens to make the cache invalidation worthwhile", and S1 sets it to 4,000 input tokens. In measured run s1-1, the first clear came at request 22: it removed 19 tool results, about 5,012 input tokens, and that request read only 1,446 tokens from the cache and wrote 5,369. The next requests read the cache again, until the next clear.

S2: notes, compaction on demand and the swap

S2 is S1 plus two mechanisms, each with its own job.

The memory tool gives the agent a directory of files that persists across requests and stays outside the window until the agent views a file. The tool is {"type": "memory_20250818", "name": "memory"}, and its commands are view, create, str_replace, insert, delete and rename (memory tool). The API defines the tool; your code runs every operation. The kit's handler, src/context/memory.ts, confines every path to the run's own memories/ folder, refuses traversal such as /memories/../../secrets.env and links that leave the folder, and caps file sizes. S2's instructions ask the agent to keep one JSON progress file, /memories/progress.json, with each ticket's state, the standing instructions in force and the decisions made. JSON follows the advice in Anthropic's harness article, where "the model is less likely to inappropriately change or overwrite JSON files compared to Markdown files" (Young, 2025). S2's clearing excludes the memory tool, so its results are never cleared.

Compaction on demand replaces the conversation with a summary when the application decides. S2 decides between tickets: when the last request carried 14,000 input tokens or more before clearing, the runner sends one extra request with the beta header compact-2026-09-04 and a top-level "compaction": {"type": "summarize", "instructions": "..."} (compaction on demand). The response is a single compaction block with a signature and stop_reason "compaction". The runner then makes the swap:

1. Put the block first in messages, and remove every message it summarizes.
2. Send exactly one block, and keep sending the header on every later request that carries it.
3. Restate each standing instruction in a role: "system" message right after the next new user turn.

The third step comes from the documentation: "role: "system" messages inside the summarized range are summarized too, so their text instructions stop applying once the block replaces them", and to keep one, "state it again in a role: "system" message" (compaction on demand). The hold on ORD-3111 is exactly such a message. S2's summary instructions also name what must survive: open tickets, standing instructions with their order ids, and every payout already issued.

The swap has two mistakes that raise no error. In the documentation's words: "If summarized messages remain after the block, the API sends them to Claude again. If a later request leaves the block out, Claude gets no summary." The spike measured both on 2026-10-29. The token counting endpoint put the conversation at 12,620 input tokens before the summary. By the usage the API returned, the next request after a correct swap carried 2,935 input tokens; with the summarized messages left in place it carried 13,988, and with the block left out it carried 1,567, because Claude saw neither the summary nor the history. Neither request returned an error. The kit's runner makes the swap one way, the block first and nothing it summarizes after it; two fault settings make each mistake on purpose, for probe P05 and the kit's test R-3, and pnpm shift:requests shows whether every later request carried the block.

The swap's other mistakes do return errors, and the spike recorded each one: a block that is not first returns compaction_block_misplaced, an edited block returns compaction_content_mismatch, and a later request without the beta header returns a validation error that does not mention the header. Two blocks in one request returned compaction_block_ambiguous, an error the documentation did not list when it was read. And a summary request whose instructions invite a tool call can come back with stop_reason "tool_use" and no block at all, which is why the runner checks stop_reason before it looks for the block.

What compaction costs. The summary is one more call: "The summarization call uses the request's model, system, tools, thinking settings, and max_tokens", and its usage appears as a compaction entry in usage.iterations. The documentation is direct about counting it: "To count what a conversation consumed, sum across usage.iterations, not the top-level fields." The kit's cost estimate does that. The top-level fields alone under-count every S2 run, and Measuring the strategies shows by how much.

Reading a context inventory

A token count says how much a request carried. An inventory says what. The kit's inventory command reads a run record and lists, for every request, each piece of the window: its kind, where it came from, its version or the time it was read, an estimated token count, and whether the shift's records make it stale at that moment. Stale is judged from the shift's ground truth (the policy library's effective times and the order events), never from what the agent believed.

pnpm shift:replay --run runs/measured/s0-1.jsonpnpm shift:inventory --run results/replay/s0-1

The token figures are estimates. Each piece is counted with Anthropic's free token counting endpoint, as a message on its own minus a one-character message, and the endpoint says "The token count is an estimate". The request totals come from the usage the API returned. The two are different measurements, so the inventory prints both and never claims they match. The download ships the counts for every recorded piece in data/token-cache.json, so the inventory of a replay makes no call.

Here is S0 in run s0-1, at five requests: the first, the first after finance refunded ORD-3104 at 10:05, the first after the policy changed at 10:25, the first for T-2109, and the last.

Read it from left to right. The "Not needed by any ticket" column counts what no ticket in the shift ever uses: 102 orders that no ticket touches, 8 reference documents and 80 earlier closed tickets. That was 86 percent of the first request's estimated input and 72 percent of the last request's. The "Stale" column starts at zero and becomes nonzero twice. From request 14, S0 carried the 09:00 copy of ORD-3104, which no longer matched the order after the 10:05 refund; from request 17, it carried version 4 of POL-DOA-30 beside version 5. By T-2109, the ticket the stale copy could mislead, the window held both stale pieces.

S0 still decided T-2109 correctly in all five runs, because an operator notice in the conversation said the order had been refunded. The inventory shows the risk; the checks show the outcome. Probe P04 in Failure gallery removes the notices and shows what the stale copy does on its own.

S1 and S2 look different. S1's largest request in run s1-1 carried 15,158 input tokens by its usage, and by its last request clearing had replaced 15,812 tokens of old tool results with placeholders. S2's last request in run s2-1 carried a compaction summary, the restated hold and its recent tool results, 11,426 input tokens by its usage.

Fetching did not keep those windows free of stale pieces; it added a fresh copy beside the old ones. After 10:25, the version 4 text of POL-DOA-30 in earlier tickets' search_policy results stayed until clearing or a summary removed it: in s1-1 for 6 requests (29 to 34), up to 2 copies at a time, and in s2-1 for 5 requests (31 to 35), up to 2 copies at a time. Each ticket fetched the policy again, so version 5 sat beside it, and the checks show that the agent applied version 5. In s1-1, the get_order result for ORD-3104 that T-2104 read at 09:45 had been cleared before finance's refund at 10:05 made it stale, so Claude never saw a stale copy of that order. The inventory counts a cleared result in its "Cleared" column, never as stale.

Measuring the strategies

A comparison is only as good as its protocol, so the protocol came first and the code that applies the decision rule was written before the measured runs.

The protocol

• The workload: the shift from seed 8 (data/shift/SHIFT-HASH.txt), the three strategy files with their hashes in every run record, @anthropic-ai/sdk 1.3.0 (API version 2026-10-21), and claude-sonnet-5-5 at effort medium. S0's requests carry no beta header, S1's carry context-management-2025-06-27, and S2's carry it with compact-2026-09-04, which each compaction request carries alone.
• Repeats and order: five runs per strategy, in a fixed rotation so that no strategy always ran first or last: S0, S1, S2, S1, S2, S0, S2, S0, S1, S0, S1, S2, S1, S2, S0. Each run's system prompt opens with its run id, so no run could read another run's cache.
• Quality first: every check on every ticket of every run is reported, with n.
• Then tokens, cost and time, as the median and the full range, and only between strategies whose checks agree.
• Cost is an estimate: each response's usage, every iteration summed, at the list prices read on 2026-10-29, with cache writes and reads at their own rates. It is not an invoice.
• The decision rule: rank by checks. Tokens, cost and time decide only between strategies whose checks agree, and one strategy uses fewer tokens or costs less only if its range sits below the other's in every run. Keep the more complex strategy only when it meets that bar; otherwise return to the simpler one. A run that loses the hold, applies a superseded policy or pays from a stale balance fails C-HOLD, C-POLICY or C-FRESH, so its strategy ranks below any strategy that passed them.

Results

The runs ran on 2026-10-29. The table is what pnpm shift:report --runs runs/measured prints from the recorded runs in the download.

Extended description

A table-like chart with three rows and two measures. Column "Every check": S0.v1 (load everything, 5 runs), S1.v1 (retrieve and clear, 5 runs) and S2.v1 (notes and compaction, 5 runs) each passed 60 of 60 ticket-runs. Column "Median input per request", with a bar for each: S0.v1 45,297 (45,134 to 45,422), S1.v1 9,824 (9,607 to 9,902), S2.v1 10,279 (9,766 to 11,131). Each bar is the median, across the five runs, of each run's median input per request, and a mark at its end spans the lowest and highest run. Input per request counts compaction requests too, as the kit's report does. The footer reads "Estimates from usage, runs of 2026-10-29". The caption adds the model, claude-sonnet-5-5 at effort medium, and @anthropic-ai/sdk 1.3.0.

Every strategy kept every decision right. Each passed every check on all 60 of its ticket-runs: the hold on ORD-3111 held at T-2111, T-2109 went to a human with no payment, every damage refund after 10:25 followed version 5, and no ticket was paid twice. On this shift, task quality does not separate the strategies.

So tokens, cost and time decide, and they point different ways. S1 used the fewest input tokens and cost the least in every run: its most expensive run, $0.2650, cost less than S0's cheapest, $0.3100. S0 was the fastest, at a median 79.4 s of model time against S1's 104.4 s, because it made 29 requests per run where S1 made 52 or 53. S2 cost the most and took the longest, and its peak request was larger than S1's in every run: 16,818 to 18,616 input tokens, against 14,865 to 15,567. Two records explain it. Every S2 request starts with a larger fixed part, because the memory tool brings its definition and the protocol the API adds: the first request of each S2 run wrote 3,354 tokens to the cache against S1's 1,560. And S2 writes: its median run produced 19,282 output tokens against S1's 7,400, because every progress-file update is a tool call whose content Claude writes, billed as output at $10 per million tokens, and every summary is output too.

Caching is why S0 was not the most expensive. S0 sent about 1,314,710 input tokens per run against S1's 489,419, but 96 percent of S0's input was read from the cache at $0.10 per million tokens (pricing). S1's clearing gave up some of that discount: when the set of cleared results changed to one the cache did not hold, the prompt was written to the cache again from the first cleared result. In run s1-1, what clearing removed changed on 3 requests: 22, 35 and 48; across S1's runs, on 3 to 7 requests per run. Between those points, S1's requests read the cache like any other.

The top-level fields under-count compaction. In every S2 run, the top-level usage fields left out the summary requests: the cost estimated from them was $0.0552 to $0.1281 lower than the estimate summed across usage.iterations, 15 to 25 percent of the full estimate.

Applying the rule. The checks agree, so the rule compares cost: S1 sits below S0 and S2 in every run, so S1 is the strategy to keep for this shift, at the price of 25.0 s more model time than S0 at the median. S2 does not meet the bar against S1 on any measure, so the rule returns from S2 to S1.

What this does not show

• One shift, one model, one effort. Twelve tickets, seed 8, claude-sonnet-5-5 at medium, five runs per strategy. S0's largest request was 50,291 tokens, about 5 percent of the 1M-token window. A longer queue, a larger library or a different model could change every row, and this guide does not extrapolate to them.
• Whether S2 earns its cost on a longer shift. S2 exists for conversations that would outgrow the window or degrade before they end. This shift never got there, so the measurement says nothing about that case.
• What the claim grader cannot see. The claim check is a model grading a reply against a digest of the records, and the digest decides what it can check. During these measurements, the first digest made the grader fail true statements that the digest left out, such as the hold at T-2111 and the amount the customer stated at T-2109. Each fix added records, not leniency, and was tested with control replies before every run was graded again: eleven controls, true and false, of which the final digest got ten right. The one it missed misquoted the amount the customer stated, so the claim check can pass a reply that misquotes what the customer asked for. The final digest still leaves out other tickets' outcomes and the agent's own notes, so a true statement about either can fail; Lab 2 reads two such failures. The measured records list every regrade and every verdict that changed.
• Prices. Costs are estimates at the list prices read on 2026-10-29. Prices change; the run records keep the usage, so the cost can be recomputed.

What the runs did not show

Three failures this guide set out to show did not happen in any recorded run. Their counts:

• A superseded policy applied. In 89 ticket-runs across the recorded runs, the window carried both version 4 and version 5 of POL-DOA-30 while the agent worked a ticket under that policy, and C-POLICY passed in all 89. S0 and P04 carry every version in the system prompt, so after 10:25 both versions sat in every request; the other strategies carried version 4 in old tool results until clearing or a summary removed them.
• The hold lost after compaction. Of the recorded runs, 13 compacted, 40 times in all. In every one, T-2111 went to a human with a note naming the hold on ORD-3111, including the two S2.v2 runs, which do not restate it, and the two P05 runs, which leave the block out.
• A role: "system" instruction lost in the summarized range. Of those, 7 compacted without restating the hold after the swap: P01's 3, with the default summary prompt and no notes; P05's 2, which leave the block out; and the 2 S2.v2 runs. T-2111 went to a human with a note naming the hold in all 7. In P01 and S2.v2 every summary named the hold on ORD-3111; in P05, whose later requests carry no summary, the agent read the hold from its progress file. So on this shift the documented risk did not turn into a failure. That is no reason to skip the restatement: it is what the documentation says to do.

Labs

The labs are in the download, context-engineering-labs.zip, with the handout labs/learner-labs.md. Each lab has a live route, which calls the Claude API with your key and spends a little, and an offline route, which replays the runs recorded on 2026-10-29 and spends nothing. The times are planning estimates; no learner pilot has timed them yet.

On the live route the labs cost roughly $2.11 in API usage at 2026-10-29 list prices, by the costliest recorded run of each kind; the offline route replays recorded runs and costs nothing.

Each amount is the costliest completed run with the same settings, among the recorded runs and the live runs of the clean-room walk, plus the costliest claim grading of one run, at the list prices of 2026-10-29. A run of yours can cost more. Every live run stops at its spend cap, $1.50 by default, checked after each response.

Troubleshooting

Each entry quotes what the kit printed when the clean-room walk of the download took that path on 2026-10-29, then gives the fix and the check that shows it worked.

T-01: pnpm is not found

bash: line 1: pnpm: command not found

Fix: On Node.js 22 or 24, run corepack enable, then pnpm --version; on Node.js 25 and later, install pnpm 12 as the pnpm installation page describes. Check: pnpm --version prints 12, and pnpm shift:preflight reports it.

T-02: A live run with no key

ANTHROPIC_API_KEY is not set. Live runs need a key with a spend limit, or use the offline replay (pnpm shift:replay), which makes no model call.

Fix: Set ANTHROPIC_API_KEY in the shell for a key that has a spend limit, or take the offline route, which replays the recorded runs. Check: pnpm shift:preflight says a key is present; it never prints the key.

T-03: A run id that is taken

Run id s0-mine is taken (<download>/labs/starter/results/shift/s0-mine). Choose a new one; runs never overwrite.

Fix: Give the run a new --run-id. Runs never overwrite, so every record you graded stays as it was. Check: the run prints its records folder.

T-04: The shift does not match the recorded runs

The recorded run used shift 7e6d415a6d62; <download>/labs/starter/data/shift holds 3ef76a80aaf0. Regenerate the shift with seed 8.

Fix: The recorded runs used the shift from seed 8. Run pnpm shift:generate --seed 8. Check: data/shift/SHIFT-HASH.txt starts with 7e6d415a6d62, and the replay prints REPLAY PASS.

T-05: The inventory has no count for some pieces

31 piece(s) of this run are not in the token cache. Run again with --counts live (free, needs a key), or use a run whose counts ship with the kit.

Fix: A run you recorded yourself has pieces the shipped token cache does not hold. Run the inventory with --counts live: the token counting endpoint is free, but it needs a key. The new counts go to results/token-cache.json, and the shipped data/token-cache.json does not change. On the offline route, read the inventory of a recorded run instead. Check: the inventory writes inventory.md.

T-06: The request list is given a bundle

No calls.jsonl in runs/probes/p05-1.json. Give a run folder; to read a bundle, replay it first: pnpm shift:replay --run runs/measured/s1-1.json

Fix: A bundle in runs/ keeps a digest of each request, not the request. Replay it first, then give shift:requests the replay's folder. Check: the list prints one row per request.

T-07: A prediction that already exists

predictions/S1.v2.json exists. A prediction is written once, before the runs; to predict a changed strategy, give it a new version.

Fix: A prediction is written once, before the runs. To predict a changed strategy, give the strategy a new version and predict that. Check: the new file appears in predictions/.

T-08: The report check fails

PASS  every field filled: 18 fieldsPASS  runs folder holds comparable graded runs: 21 comparable of 21 graded run(s) in runsPASS  every strategy in the runs is compared: P04.v1, S0.v1, S1.v1, S1.v2, S2.v1, S2.v2PASS  P04.v1: runs: report 2, runs 2PASS  P04.v1: ticket-runs passing every check: report 22 of 24, runs 22 of 24PASS  P04.v1: every failed check reported: report "C-OUTCOME 2, C-FRESH 2, C-CLAIMS 2", runs "C-OUTCOME 2, C-FRESH 2, C-CLAIMS 2"PASS  P04.v1: input tokens per run (median, lowest, highest): report "1,364,064 (1,360,385 to 1,367,743)", runs 1364064 (1360385 to 1367743)PASS  P04.v1: estimated cost per run (median, lowest, highest): report "$0.3251 ($0.3221 to $0.3282)", runs $0.3251 ($0.3221 to $0.3282)PASS  S0.v1: runs: report 5, runs 5PASS  S0.v1: ticket-runs passing every check: report 60 of 60, runs 60 of 60PASS  S0.v1: every failed check reported: report "none", runs "none"FAIL  S0.v1: input tokens per run (median, lowest, highest): report "1,314,000 (1,308,535 to 1,316,840)", runs 1314710 (1308535 to 1316840)PASS  S0.v1: estimated cost per run (median, lowest, highest): report "$0.3156 ($0.3100 to $0.3166)", runs $0.3156 ($0.3100 to $0.3166)PASS  S1.v1: runs: report 5, runs 5PASS  S1.v1: ticket-runs passing every check: report 60 of 60, runs 60 of 60PASS  S1.v1: every failed check reported: report "none", runs "none"PASS  S1.v1: input tokens per run (median, lowest, highest): report "489,419 (473,768 to 494,708)", runs 489419 (473768 to 494708)PASS  S1.v1: estimated cost per run (median, lowest, highest): report "$0.2390 ($0.2318 to $0.2650)", runs $0.2390 ($0.2318 to $0.2650)PASS  S1.v2: runs: report 2, runs 2PASS  S1.v2: ticket-runs passing every check: report 23 of 24, runs 23 of 24PASS  S1.v2: every failed check reported: report "C-CLAIMS 1", runs "C-CLAIMS 1"PASS  S1.v2: input tokens per run (median, lowest, highest): report "597,922 (585,276 to 610,568)", runs 597922 (585276 to 610568)PASS  S1.v2: estimated cost per run (median, lowest, highest): report "$0.2418 ($0.2224 to $0.2612)", runs $0.2418 ($0.2224 to $0.2612)PASS  S2.v1: runs: report 5, runs 5PASS  S2.v1: ticket-runs passing every check: report 60 of 60, runs 60 of 60PASS  S2.v1: every failed check reported: report "none", runs "none"PASS  S2.v1: input tokens per run (median, lowest, highest): report "600,612 (565,516 to 640,578)", runs 600612 (565516 to 640578)PASS  S2.v1: estimated cost per run (median, lowest, highest): report "$0.4228 ($0.3766 to $0.5213)", runs $0.4228 ($0.3766 to $0.5213)PASS  S2.v2: runs: report 2, runs 2PASS  S2.v2: ticket-runs passing every check: report 23 of 24, runs 23 of 24PASS  S2.v2: every failed check reported: report "C-CLAIMS 1", runs "C-CLAIMS 1"PASS  S2.v2: input tokens per run (median, lowest, highest): report "580,980 (578,977 to 582,982)", runs 580980 (578977 to 582982)PASS  S2.v2: estimated cost per run (median, lowest, highest): report "$0.3934 ($0.3903 to $0.3965)", runs $0.3934 ($0.3903 to $0.3965)PASS  the recommendation names a compared strategy: S1.v1PASS  the recommendation is not outranked on task checks: S1.v1 has the fewest check failures (none)1 of 35 check(s) failed.

Fix: Each line names a figure in your report that does not match the runs it cites. Copy the figure from pnpm shift:report for those runs, or cite the runs it came from. Check: pnpm shift:report-check prints PASS.

T-09: A strategy file the kit will not load

strategies/S1.v3.json is not a valid strategy file:clearing.trigger_input_tokens: Invalid input: expected number, received string

Fix: Each line names a field and what is wrong with it. Ids are S and a digit, or P and two digits, and the file name must be the id and version, such as S1.v2.json. pnpm shift:run refuses the same file with the same message, printed above a stack trace, so check a new file with shift:diff before you run it. Check: pnpm shift:diff loads the file and lists what differs from the version it came from.

T-10: A run that stops at its spend cap

s1-cap T-2101 (09:00)s1-cap T-2102 (09:15)Run s1-cap: spend_cap_run; 6 requests; 0 compaction(s); estimated $0.0203 (top-level fields only: $0.0203).Stopped at the spend cap; the last response went $0.0003 past it.Records: <download>/labs/starter/results/shift/s1-cap

Fix: The run stopped after the response that crossed --cap-run, so the tickets it did not reach have no outcome, and the grade says the run is not comparable. Raise the cap if your spend limit allows, or take the offline route. Check: the next run prints completed.

Quick reference

Read in Anthropic's documentation and tested on @anthropic-ai/sdk 1.3.0 on 2026-10-29. Context editing and both kinds of compaction are in beta.

The swap, every time. Put the block first and remove what it summarizes; send exactly one block; keep the header on every later request that carries it; restate each standing instruction in a role: "system" message after the next new user turn; check stop_reason before looking for the block.

Counting what a run used. Sum every entry of usage.iterations, not the top-level fields, and price uncached input, cache writes and cache reads at their own rates. On Sonnet 5.5, per million tokens on 2026-10-29: $2 input, $2.50 for 5-minute cache writes, $0.10 for cache reads, $10 output (pricing).

Commands from the kit, run from labs/starter:

pnpm shift:preflight                                         # versions, shift hash, strategies, recorded runs, key present or notpnpm shift:run --strategy strategies/<ID>.v<N>.json --run-id <id> [--model scripted]pnpm shift:grade --run results/shift/<id> --claims model     # or --claims rules (offline), --claims recorded (a replay)pnpm shift:replay --run runs/<set>/<id>.json                  # offline: rebuild a recorded run, no model callpnpm shift:inventory --run <run folder> [--counts live]      # what each request carried, by kind, and what was stalepnpm shift:requests --run <run folder>                       # each request's usage, clears and compaction blockpnpm shift:diff --a <strategy or bundle> --b <strategy or bundle>pnpm shift:predict --strategy strategies/<ID>.v<N>.json      # a stamped prediction, before the runpnpm shift:report --runs <folder> [--predictions predictions]pnpm shift:report-check --report <report.md>