Ask a consultant how to get value from AI and you’ll be told to write better prompts. Ask them in eight months and the prompts won’t work the same way, because the model behind them will be gone.

Don’t build around the prompt. Build around the twenty examples you can run it against.

Why the models keep moving

Anthropic is direct about this. It retires older models “to ensure capacity for new model releases”, and requests to a retired model fail outright. It gives at least 60 days’ notice by email. Its own documentation advises testing your application against replacement models well before that date, and warns that deprecated models are less reliable than active ones.

Read that as a business owner rather than a developer. Your assistant worked on Friday. On Monday the integration errors. A restart won’t fix it. Someone has to re-evaluate, because the new model answers differently, and “differently” isn’t the same as “better”.

Google’s price doubling on December 1 pushes the same direction from a different angle. Cheaper models get better, prices move, and the model you benchmarked by hand last spring isn’t the model you’d choose now.

What “it looks good” is worth

The normal process goes: try ten examples, they look right, put it into service. That’s a reasonable way to decide whether a tool is interesting. It’s a bad way to decide whether a process is safe to run without someone watching it.

These models fail unevenly, and that’s the problem. Chroma’s testing across 18 models found accuracy dropping as inputs grew, and the drop wasn’t smooth. Some models held steady and then fell off a cliff. What separated a good answer from a bad one was often the presence of similar-looking wrong material sitting near the right answer, not the length of the input.

That’s the failure mode that matters commercially. Not nonsense. Something that reads correctly and is subtly wrong, on the case that looks like the other nine but isn’t.

You can’t see this by eyeballing a demo. You can only see it by scoring a fixed set of cases.

The asset, in plain terms

A test set is a folder. Inside it:

  • 30 to 50 real examples from the last few months. Real inputs, not invented ones. Include the awkward ones, because the awkward ones are what the process will actually meet.
  • The correct answer for each. Written by whoever currently does the job. This is the part that takes effort and the part nobody does.
  • A way to mark pass or fail. For sorting and extracting, a comparison does it. For anything written, it’s one person spending two minutes per example.

That’s the whole thing. No software, no dashboard.

Now every decision becomes cheap. New model? Run the set, get a number, compare it to last time. Vendor raises prices and offers a cheaper tier? Run the set. Someone says a new model will halve your costs? Run the set before you believe them, not after.

The set is also the thing your team keeps. Prompts get rewritten. Models get retired. Vendors get acquired. The document containing 40 real cases and what right looks like is worth more every year, and it’s the reason a business can change suppliers in an afternoon instead of over a quarter.

What to do this month

Pick the one AI task you rely on most. Get whoever does it properly to write down 30 real examples and their correct answers. Budget two hours for that, and count it in the project cost.

Then run your current setup against it and write the number down.

That number is the first honest thing most businesses have ever said about their AI. Everything after it, including every vendor conversation, gets decided against it.