The sales pitch for a million-token context window is hard to argue with. Stop choosing what to give the AI. Give it everything: the whole policy manual, three years of emails, every contract. It can hold all of it, so it will consider all of it.

The research says otherwise, and the bill says it twice.

What the tests found

Chroma’s research team ran this properly. They tested 18 current models on tasks deliberately kept simple, so that the only thing changing was how much irrelevant text sat around the question. Repeating a list of words. Answering a question about one sentence buried in a document.

Accuracy fell as input grew. Not gently, and not evenly. Some models held up until a point and then dropped sharply. Others degraded gradually. On a task where the correct answer was effectively memorization, more context made the models worse.

Two findings matter more than the headline:

The type of filler matters. Text that’s related to the question hurts far more than text that’s obviously irrelevant. Padding with unrelated material is nearly harmless. Padding with similar-looking-but-wrong material is what breaks it. Researchers call these distractors, and they describe them as topically related to what’s being asked but not actually answering it.

Models are bad at noticing what’s missing. A separate benchmark, AbsenceBench, tests whether a model can tell you that something isn’t in a document. Accuracy falls as the document grows. So the model that confidently summarizes 200 pages of policy is also the model least able to tell you the one clause you needed is absent.

Why this costs you money twice

Every token you send is billed. Sending the whole manual instead of the four pages that apply costs the difference on every single request, forever.

Then it costs you again in something harder to see: the answer got worse, and nothing told you. A short, targeted prompt that gets it right beats a long, complete prompt that gets it wrong. The long one feels more thorough. It’s also more likely to hand you a confident answer drawn from the third-most-relevant document in the pile.

What to do instead

You don’t need retrieval infrastructure or a vector database. You need three habits.

Ask a question, then feed the answer. “Which of these five documents covers supplier payment terms?” costs almost nothing. Then send only the document it names. Two cheap steps beat one expensive guess.

Keep the pile small on purpose. If a task needs 400 pages, ask what the task actually needs. Usually it needs a section, a template, or the last decision made about the subject.

Make absence explicit. Ask for the source of every claim, and ask what the model couldn’t find. If it can’t cite where a statement came from, treat it as unverified rather than as an answer.

The honest version

Long context is genuinely useful when the task really is “read all of this and hold it in mind at once”, and there are such tasks. It’s a poor substitute for knowing which documents matter.

Most small businesses don’t have a knowledge problem. They have a filing problem, and a bigger pile makes a filing problem worse rather than better. If the AI can’t find the right page, adding nine more pages won’t fix it. It’s a larger haystack and a higher bill.

Before you build anything that feeds documents to a model, count the pages and ask which ones earn their place. That question takes an hour, and it’s where the savings are.