Guide

Why AI agents drift: attention decay and context window limits

Published 23 September 2026

An agent that followed your rules cleanly for ten turns starts cutting corners by turn forty. The instruction did not change, and the model did not get worse between those turns. What changed is how much other material now sits between that instruction and the token being generated right now.

That is drift, and the popular explanation of it points at the wrong fix.

Drift is usually competition, not forgetting

Nothing was deleted. The constraint you wrote at turn one is still there at turn forty, one instruction among a much larger pile of tool definitions, tool results, retrieved documents, intermediate reasoning and forty turns of conversation, all of which are also asking for attention.

Anthropic's Applied AI team ties this to the architecture in "Effective context engineering for AI agents". Transformers let every token attend to every other token across the entire context, which "results in n² pairwise relationships for n tokens". From that comes the frame that matters: "LLMs have an 'attention budget' that they draw on when parsing large volumes of context." Every new token depletes some of it, so context "must be treated as a finite resource with diminishing marginal returns".

There is no quality wall at a particular token count. Each token simply buys less than the one before it, until the extra material costs more in dilution than it adds.

The window fills faster than most people assume

Anthropic's Claude Platform documentation on context windows is specific about what counts toward the window:

  • the system prompt
  • every message including tool results
  • images and documents
  • the tool definitions themselves
  • generated output including extended thinking

That is why an agent fills a window far faster than a chat does. A single tool call can return thousands of tokens of JSON you never read and cannot remove.

The documentation is also blunt about the tradeoff: "more context isn't automatically better", and "As token count grows, accuracy and recall degrade, a phenomenon known as context rot."

The same documentation puts current windows at 1M tokens for a number of models and 200k for others including Claude Sonnet 4.5. A bigger window buys room, not uniform attention across that room.

What the research actually measured

Two studies carry the evidence here. Chroma's context rot report found that performance degrades as input length grows, measured across 18 models, rather than holding uniform. Liu et al.'s Lost in the Middle paper found a U-shaped curve, with performance highest when the relevant information sits at the very beginning or the very end of the input.

Chroma's context rot report. Kelly Hong, Anton Troynikov and Jeff Huber published "Context Rot: How Increasing Input Tokens Impacts LLM Performance" on July 14, 2025, evaluating 18 models across the Claude, OpenAI, Gemini and Qwen3 families. Their starting point is that models are usually presumed to process context uniformly, and that in practice this assumption does not hold. Four results matter for drift:

  • As needle-question similarity decreases, "model performance degrades more significantly with increasing input length".
  • Distractors compound. "Even a single distractor reduces performance relative to the baseline (needle only)", and different distractors hurt by different amounts.
  • Structure works against you: "models perform worse when the haystack preserves a logical flow of ideas". Shuffling the haystack improved performance across all 18 models.
  • On a trivial word-replication task, accuracy was highest when the unique word sat near the beginning; failure modes included refusal (2.89% for Claude Opus 4), invented words absent from the input, and under-generation.

The cleanest demonstration is their LongMemEval result: a focused prompt averaging roughly 300 tokens beat the full prompt averaging roughly 113k tokens containing the same relevant information. "Across all models, we see significantly higher performance on focused prompts compared to full prompts."

Chroma varied input length, not conversational age. A long conversation is one way input gets long, so the finding transfers, but this is not a study of multi-turn agents. Chroma is also widely credited with coining the phrase "context rot", which the report neither claims nor cites prior use for.

Lost in the middle. Liu et al. found a U-shaped curve: performance is highest when the relevant information sits at the very beginning or the very end of the input, and degrades significantly when the model must reach into the middle. Two caveats it is routinely cited without. GPT-3.5-Turbo's multi-document QA performance "can drop by more than 20%", which is one model on one task, and a relative drop rather than twenty percentage points. And the models tested were 2023-era, including GPT-3.5-Turbo and Claude-1.3, so it is evidence for the shape of the effect, not a measurement of today's frontier models.

The consequence for how you write constraints

Do not rely on the model to remember. Front-load constraints as hard rules, and attach a reason to each one. The reason is not decoration: a bare prohibition has to be recalled verbatim to be applied, while a rule with a stated reason can be re-derived from the situation in front of the agent. Chroma's needle-question similarity finding points the same way on wording: state the rule in the vocabulary of the situation that should trigger it, not in abstract policy language.

The other half is subtraction. Anthropic's post puts the goal as "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome". Every near-miss document you leave in context is a distractor competing with the thing you actually need.

The three real mitigations, and what each costs

Anthropic's post describes three techniques for long-horizon work. None of them is free.

Technique What it does What it costs you
Compaction Summarizes a context window nearing its limit and reinitiates a new window with the summary Lossy by design. Whatever the summary drops is gone
Structured note-taking Persists notes outside the context window, giving what Anthropic calls "persistent memory with minimal overhead" The agent has to choose to write them and choose to read them back
Sub-agents Specialists work in clean context windows, each exploring with tens of thousands of tokens or more and returning a distilled summary often 1,000-2,000 tokens long The coordinating agent sees the summary, never the evidence behind it

That Anthropic ships compaction at all is a useful signal that the degradation is treated as real rather than as a benchmark artifact. Anthropic's compaction documentation states the rationale plainly: "as a conversation grows, response quality degrades, so compaction replaces older content with a concise summary." In beta the trigger defaults to 150,000 input tokens with a minimum of 50,000, and it accepts custom instructions for the summarization prompt. That last option is where you make sure your non-negotiable constraints survive the compaction boundary rather than being summarized away.

Where this fits if you use Bespoke Prompting

The Agent build type in Bespoke Prompting's spec generator emits a GUARDRAILS section where every rule takes the form "Never X because Y", covering at minimum data privacy, scope, impersonation and irreversible actions, each specific to that agent's own domain and tools. One of the engine's real examples: "Never log full customer records because they contain PII subject to retention limits." The because clause gives the rule something to be re-derived from at turn forty, when the original instruction is buried under everything that arrived since.

Sources

Suggested internal links