Tamarillo Research
An agent's history grows with every plan, tool call, and observation. Deleting some of it seems like an obvious way to make the next call cheaper. But a shorter prompt can require more computation: an edit may destroy reusable work, and the agent may spend its next calls reconstructing what it has forgotten.
We account for both effects by pricing the history an agent carries, the work an edit forces the server to rebuild, and the different continuation the edit induces.
One result captures why this matters. On the same 103 recorded Qwen3-8B ScienceWorld pairs, dropping earlier reasoning costs 1.74 times keeping it when prompt and generated states are reusable. With no reuse, the ratio is 0.34. The recorded continuations and task outcomes do not change between those price calculations. The serving rule changes which policy looks efficient.
A cached token still has a carrying cost
A prefix cache lets a server reuse computations from an earlier call. If the next prompt extends a cached prefix, the engine can avoid processing those positions again. But each new query still attends to the retained keys. Remembering avoids some processing; it continues to incur attention on later work.
We describe a run with two counts: $N$, the number of token positions processed, and $Q$, the number of token pairs attended. A position processed twice counts twice. At the model's declared architectural prices, the reduced cost is:
$$C = cN + \beta Q.$$
Here $c$ prices a token's projection work and $\beta$ prices an attended pair. To account for generation being sequential and often more expensive on the serving hardware, we add a decoding premium $w$:
$$C_{\rho,w} = cN_\rho + \beta Q_\rho + (w-1)cG.$$
$G$ counts generation positions. The cache rule $\rho$ distinguishes reuse of prompts and generated states from reuse of prompts alone. Actual reuse also depends on exact token matches, cache-block alignment, and eviction.
The ratio $P^* = c/\beta$ gives an architectural exchange rate: the context length at which a position's attention cost equals its projection cost. In this reduced account, it is about 124K tokens for Qwen3.6-27B and 30K for a synthetic counterpart with all layers using full attention. The latter is a pricing construction, not another trained model or a measured behavioral comparison.
These are operation prices. They are not an API tariff or a prediction of elapsed time.
Why deleting reasoning can rebuild the answer
Consider a cached 16,000-token task followed by 2,000 reasoning tokens and a 100-token action. The next call receives 500 new input tokens and evaluates 100 generation positions.
With generated-state reuse, keeping the history lets the server reuse the reasoning and action. Dropping the reasoning changes the prefix before the action, so the surviving action must be processed again. With prompt-only reuse, keeping also requires processing the returning reasoning. The same deletion now avoids that work.
This example uses Qwen3.6-27B operation prices, equal input and generation base prices, token-granular reuse, and no eviction. Block widths are schematic.
The size of the text after the cut matters. For a reasoning block of $r$ tokens after a prefix of $H$ tokens and before an answer of $a$ tokens, the retention horizon under generated-state reuse is:
$$L^* = \frac{a}{r}\left(P^* + H + \frac{a+1}{2}\right).$$
$L$ counts later evaluated positions common to the two continuations. Below this horizon, carrying the reasoning is mechanically cheaper than rebuilding the answer. Above it, deletion is mechanically cheaper, under the stated exact-prefix assumptions.
For the Qwen3.6-27B reference at a 20K-token prefix, a 2,000-token trace ending in a seven-token answer is worth keeping for roughly 500 more evaluated positions. If it ends in a 4,000-token command, that horizon is about 290K. The reasoning block has the same length; the cost of rebuilding what follows it is very different.
Small edits can create large rebuilding bills
Replacing a block with a shorter summary has three mechanical effects: producing the summary costs work, the replacement and surviving tail must be rebuilt, and later queries attend to fewer keys. An edit pays for itself when accumulated attention savings recover its production and rebuilding costs.
That makes timing consequential. We replayed 132 fixed continuations, preserving every generated response, action, and external input while changing only the reasoning-deletion schedule. A sliding window that kept the latest five reasoning blocks shortened submitted prompt volume by 1.6%, yet cost 3.97–4.56 times keeping everything across the tested price sweep with generated-state reuse. Each advance invalidated another surviving tail.
The left panels isolate deletion schedules, without replacement digests or trained editing behavior. The right panel prices an additional generated token returned in the next prompt. Shared generation cancels from the schedules' absolute cost differences.
Immediate deletion stayed within 2% of Keep under generated-state reuse throughout the sweep. Without prefix reuse, all four deletion schedules were cheaper than Keep. These findings concern the specified schedules on recorded work; they do not establish the performance of a complete trained context-management method.
The server can reverse the ranking
We also repriced the two recorded live continuations from each of the 103 matched Qwen3-8B ScienceWorld forks. At equal input and generation base prices, the aggregate results were:
| Reusable computation | Total Drop / Keep cost | 95% interval |
|---|---|---|
| Prompt and generated states | 1.74× | 1.46–2.08 |
| Prompt only | 1.02× | 0.86–1.23 |
| None | 0.34× | 0.28–0.42 |
Curves reprice the same recorded continuations at different architectural exchange rates and decoding premiums. Shading shows pointwise 95% family-bootstrap intervals; the dotted line marks the Qwen3-8B exchange rate.
The prompt-only interval spans equal cost. The full-reuse and no-reuse intervals lie on opposite sides of it. A higher decoding premium favors Keep in this panel because Drop generated more output.
Forgetting changes what happens next
Mechanical replay answers what a specified continuation costs under another policy. A live intervention answers what the agent does when shown another history. We separate the total cost change into these two parts:
$$\Delta C = \Delta M + \Delta B.$$
$\Delta M$ changes how the same continuation is processed. $\Delta B$ changes the continuation while keeping its processing policy fixed. An intervention reduces total cost when its continuation saving exceeds its mechanical expense. The allocation between the two terms depends on which continuation is used as the replay reference; their sum does not.
Task success must accompany that bill. On HotpotQA, dropping earlier reasoning reduced the full-answer-score rate by 9–23 percentage points across four models, at equal or lower aggregate operation cost. The evaluation grades answers, without a supporting-fact criterion. On ScienceWorld, the direction differed: gpt-oss-20b solved 20 of 88 tasks under Drop versus 3 of 88 under Keep. Other ScienceWorld comparisons did not establish a clear success advantage for either policy.
History carried between tasks showed another tradeoff. Across 32 matched five-task chains, carrying history under Keep cost 0.60 times fresh sessions over the whole chain. On the three subsequent new tasks in each chain, it solved 17 of 96, against 30 of 96 with fresh sessions. Spending less did not preserve the number of completed tasks.
The hosted compaction study compared automatic summaries with the released zero-shot Context Language Models (CLM) editing prompt. Across 20 completed chains in three cohorts, CLM/Summary API expenditure ranged from 0.62 to 1.79. Retention settings, tasks, and harness revisions accompanied changes in ranking. Each cell was one unrepeated chain per task stream, so these reversals are not separated from run-to-run variation. Visible THOUGHT text also remained available under both native-reasoning retention settings, unless edited or summarized.
What the evidence establishes
The accounting predictions matched the covered operation counts on 648 instrumented vLLM GPU requests, across dense and hybrid models. The validation included explicit output-head and recurrent-state terms beyond the reduced two-price account. It validates computation accounting for those executions, not a general latency or energy model.
The live comparisons have narrower limits. They condition on reaching a fork and completing the planned arm set; they are not benchmark-wide success rates. Open-model arms ran in fixed order, and action-replay resets did not verify exact environment-state equivalence. Replay reconstructs cache states from logged prompts rather than directly observing the server's cache. Models were served at different precisions. The derived retention horizon has not been validated as a deployed policy.
For context-policy comparisons, the practical reporting requirements follow directly:
- State the serving rules. Specify which earlier reasoning returns, who controls retention, and which computations can be reused.
- Record each call. Preserve prompt, cached, and completion counts, alongside rendering and cache assumptions.
- Price the complete intervention. Include replacement generation, rebuilding, and the continuation it induces.
- Report success beside cost. Keep failed attempts in the declared population and distinguish operation counts, API expenditure, and measured latency.
To transfer a policy, first reprice its recorded work under the destination system. Then run it there and measure the continuation and task outcomes it produces. The exchange rate tells us what remembering costs; the agent's behavior determines whether forgetting pays.