Product

Why each step should receive focused context.

The instinct with a large context window is to fill it. The measurements say that is a mistake twice over: retrieval accuracy falls as input grows, and several providers now charge more per token once you cross a length threshold.

By The Modeus team 2 August 2026 6 min read

A narrow illuminated channel cutting through darker terrain, used as the article's opening artwork.

The promise, and what actually happens

Context windows crossed a million tokens and the advice quietly became: stop curating, just paste it all in. It is an appealing idea. It is also the most reliably disproven claim in applied LLM work.

Three independent research efforts, using different methods and different model sets, all found the same shape: models do not use long context uniformly, and adding irrelevant material actively degrades the answer.

Position matters more than presence

The foundational result is Lost in the Middle (Liu et al., Transactions of the ACL, 2024). In a multi-document question-answering setup with 20 documents, GPT-3.5-Turbo scored 75.8% when the answer document sat first in the input, 63.2% when it sat last, and 53.8% when it sat in the middle.

The comparison that should change how you build prompts is the closed-book baseline: with no documents at all, the same model scored 56.1%. Placing the correct answer in the middle of twenty documents was worse than supplying nothing. The paper puts it plainly: "in the worst case, performance in 20- and 30-document settings is lower than performance without any input documents."

It is not just an old-model problem

A reasonable objection is that this was measured on a 2023-era model. NoLiMa (Modarressi et al., ICML 2025) tested 13 models that all advertise 128K or larger context windows, using questions whose wording does not literally overlap the target passage — so the model has to associate rather than string-match.

At just 32K tokens, 11 of the 13 dropped below 50% of their own short-context baseline. GPT-4o fell from an almost-perfect 99.3% to 69.7%. The advertised window and the usable window are not the same number.

Context Rot (Chroma, July 2025) evaluated 18 models including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 and found the same non-uniformity, plus a sharper practical finding on distractors: "Even a single distractor reduces performance relative to the baseline, and adding four distractors compounds this degradation further."

One irrelevant document is enough to measurably hurt the answer. That is the whole argument for curation, in one sentence.

Long context is also priced as a penalty

There is a mechanical reason to expect diminishing returns. As Anthropic's context engineering guidance notes, the transformer architecture means "every token to attend to every other token across the entire context. This results in n² pairwise relationships for n tokens." Their framing of the design goal is the one we build to: find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome," treating context as "a finite resource with diminishing marginal returns."

Providers have started pricing that reality directly. Google's Gemini Developer API charges Gemini 3.1 Pro at $2.00 per million input tokens for prompts up to 200K, and $4.00 above it — with output rising from $12.00 to $18.00 at the same boundary, across every service tier. Crossing the threshold doubles your input price for the same work.

What a step actually needs

The practical question is not "how much can I fit" but "what can change the outcome of this step." Those are very different sets.

Two columns contrasting working noise — unrelated workspace history, repeated instructions, generated files outside the dependency path — with decision context: current goal and definition of done, relevant files, approved constraints, and evidence needed to verify the handoff. — The split Modeus applies before a step is routed, shown on the how it works page.

Conversation history that cannot change the outcome is noise. Instructions already captured as a durable policy do not need restating every turn. Generated files outside the dependency path of the current step are not evidence, they are volume.

What survives the filter is small: the current goal and its definition of done, the files and decisions in scope, the constraints already approved, and whatever is needed to verify the handoff. Active Memory exists so that the durable half of that set — approved decisions, constraints, verified solutions — can be retrieved rather than re-sent.

Measure it before you believe it

The Modeus scenario estimator showing 20M current input tokens per month, 6M avoided repeated context, a $165.60 monthly input-cost difference and a 55% estimated input-cost reduction. — The directional estimator on Usage & Savings. It isolates two levers — context you can avoid, and remaining work that can use a lower-cost route.

The estimator is deliberately conservative about what it counts: input tokens only, excluding output, cached tokens, tool fees, subscriptions, and volume discounts. Directional numbers that you can audit are more useful than confident numbers you cannot.

Context discipline and step-level routing compound. Sending less to a correctly chosen model is where the two arguments meet.

Sources

Every figure in this article was read from the source below on 2 August 2026. Where a source carries no publication date, an access date is given instead.

Lost in the Middle: How Language Models Use Long ContextsLiu et al., Transactions of the ACL, Vol. 12 · 2024 (arXiv 2023)75.8% first / 53.8% middle / 63.2% last against a 56.1% closed-book baseline.
NoLiMa: Long-Context Evaluation Beyond Literal MatchingModarressi et al., arXiv:2502.05167 / ICML 2025 · 7 February 202511 of 13 long-context models below 50% of baseline at 32K; GPT-4o 99.3% to 69.7%.
Context Rot: How Increasing Input Tokens Impacts LLM PerformanceHong, Troynikov and Huber, Chroma · 14 July 2025Across 18 models, a single distractor measurably reduces performance.
Effective context engineering for AI agentsAnthropic Engineering · 29 September 2025n² pairwise attention; context as a finite resource with diminishing marginal returns.
Gemini Developer API pricingGoogle AI for Developers · Accessed 2 August 2026Gemini 3.1 Pro input doubles from $2.00 to $4.00 per 1M tokens above 200K.