The industry sells context length as pure progress — 200K, 1M, 2M tokens. So engineers stuff everything in "just in case." That's the bug, not the fix. Signal-to-noise beats raw volume, every time.The day a bigger prompt made my agent dumberI had a code-fixing agent that worked. Small, focused, fast. You handed it a bug, it pulled the two or three relevant files, and it shipped a patch.Then I "helped." The agent kept missing context from neighboring modules, so I did the obvious thing: I pasted in the whole repo. The full file tree. The entire conversation history from the debugging session. Every doc the retriever could find, ranked or not. Now it has everything, I thought. Now it can't miss.It got worse. Measurably worse.Accuracy dropped. Latency roughly tripled — I went from a few seconds to a coffee-refill wait per step, because every call now chewed through ~90K tokens. And the answer? It confidently edited the wrong file. The one file that actually mattered was in there — sitting somewhere around token 90,000, in the middle of a wall of imports and unrelated helpers the model glanced at and forgot.I had given it more information and less intelligence. That's not a paradox. That's how attention over long context actually behaves.A model that can read a million tokens is not a model that understands a million tokens. Capacity is not comprehension. Right where it matters most, those two things come apart.More context is not more intelligenceThe marketing frames context length as a straight line of progress. 8K, then 128K, then 200K, then a million, then two. Bigger number, better model. So the engineering reflex becomes: fill it. If the window holds 200K tokens and I'm using 6K, I'm "wasting" the model, right?Wrong, and it's the most expensive instinct in agent design right now.The window is not free storage you're obligated to fill. It's the model's working attention for this one call. Everything you put in competes for that attention. A long context doesn't give the model more focus — it spreads the same focus thinner across more tokens. You are not adding signal. You are diluting it.Two failure modes turn that dilution into wrong answers. Both are established, repeatedly-observed behavior in long-context LLMs, not speculation.Lost in the middle: present, and still ignoredTake a fact the model needs, and place it at the very start of a long prompt. The model uses it. Place the same fact at the very end. It uses it. Place it dead center, buried in thousands of tokens of other material, and accuracy on retrieving that fact drops — sometimes sharply.This is the lost-in-the-middle effect, and once you've seen it in your own logs you stop trusting "it's in the context" as a defense. Long-context models exhibit a U-shaped attention profile: strong at the edges, weak in the belly. Information at the start and end of the window gets used; information in the middle gets skimmed.That changes the whole game. The question is no longer did I include the relevant fact? It's where in the window did it land, and what did I bury it under? When I pasted my whole repo, the file that mattered didn't fall out of context. It fell into the middle of it. Same thing, from the model's point of view.So position is a design decision, not an accident of whatever order your retriever happened to return. If a fact is load-bearing, it belongs at an edge.Distractors: the model can be talked out of the right answerLost-in-the-middle is about where you put things. Distractors are about what else you put in.A distractor is a passage that's irrelevant to the question but plausible-looking — same topic, similar vocabulary, wrong answer. The intuition most engineers carry is that extra context is at worst harmless: if it's not useful, the model will just ignore it. That intuition is false. Irrelevant-but-on-topic passages actively degrade accuracy. They pull the model toward confident wrong answers it would not have given on a clean prompt.This is the part that should change how you build retrieval. Accuracy is driven by signal-to-noise, not raw volume. Ten tightly relevant chunks beat fifty chunks where forty are "kind of related." Every low-relevance passage you add is not neutral ballast — it's an active vote for the wrong answer.Which means a retriever that returns more candidates to be safe is making your agent less safe. The job isn't recall. The job is precision under a budget.Treat the window as a budget you spend, not a tank you fillHere's the mental flip. Stop thinking of the context window as storage you must fill, and start thinking of it as a token budget you spend deliberately. Every token you spend should earn its place. The default state of a token is out.So I build a context assembler that does four things on every call: retrieve candidates, rerank them by real relevance, keep only what fits the budget, and order survivors so the strongest sit at the edges. The middle gets the weakest of the keepers, because the middle is where attention goes to die.def assemble_context(query, candidates, token_budget, count): # 1. Rerank by true relevance to THIS query (a cross-encoder beats# raw vector similarity — it catches plausible distractors). ranked = sorted(candidates, key=lambda c: rerank_score(query, c), reverse=True) # 2. Greedily keep the top items that fit the budget. Drop the rest.# Dropping is the feature, not a failure. kept, used = [], 0for c in ranked: if used + count(c)