How we evaluate an AI model before betting on it

We're building a systematic editing pipeline that covers everything from typos to structural story problems. Early on we decided how the checking itself would work: one model drafts a judgment, and a second, different model checks it adversarially. The quality of that pairing determines the quality of every finding a client sees, so we don't pick it off a leaderboard — we test it, against documents seeded with known problems, and score each candidate against an answer key we wrote before any model saw the text.

First round: Claude Opus against Kimi and DeepSeek

Our first real comparison wasn't between two open models — it was against Claude Opus, our own AI provider of choice elsewhere in the business. Opus lost. Given a citation to verify, it confidently marked a nonexistent source “verified” and fabricated supporting detail for it. A far cheaper Kimi K2 Thinking / DeepSeek V4 pairing correctly flagged the same citation as unfindable. Kimi and DeepSeek became our standing pair from that point on.

Second round: chasing blind spots

Later, building a harder task — judging a manuscript's structure, argument, continuity, and pacing together — we tested that same Kimi/DeepSeek pairing against several newer models, including Gemini 3.7 Flash and Qwen3.8 Flash. Kimi/DeepSeek won again, but not cleanly: every model we tried, including our own standing pair, reliably missed a specific category of error. That's what led us to change how we tested. Instead of ranking models by one overall score, we compared them by which category each one consistently missed, then looked for a second model whose blind spot didn't overlap with the first's.

The reversal

That method turned up a pairing that covered every category cleanly on a short seeded excerpt: Gemini 3.7 Flash, alongside a second model chosen specifically to cover Gemini's one consistent gap. It looked like a clear win. Testing the same pairing against a real, full-length book — seeded with the same kind of planted issues, at real chapter length — reversed the result. The second model collapsed to single-digit accuracy at real scale, even though it had covered Gemini's gap cleanly on the short excerpt.

What we found instead

Looking at what separated the models that held up at real scale from the ones that didn't, the pattern wasn't vendor or size — it was whether a model runs an internal reasoning step before answering. Every model that used one held up on the full book; every model that didn't, didn't. Testing more reasoning-capable candidates against the same real-book fixtures turned up Qwen3.8 Flash, which beat every model in either round, at roughly a third of the cost of the pairing our short-excerpt test had recommended. Gemini and Qwen is the pairing we run today.

The lesson

This is what worked for our project. What works for someone else's will likely be different. Benchmarks are generic — they describe what a model can do in isolation, on a fixed harness, not what it will do on your system. We tested a mix of proprietary and open-weight models and settled on one of each, because that pairing worked for us, on our workload, at our budget. The lesson isn't to distrust what a vendor says about their model — or to trust what we say about ours. It's to test it yourself, on your own process, before betting the budget on it. That's more work. It means carrying some cost upfront before you know it'll pay off. But that's cheaper than shipping a bad product to a client and finding out from a complaint.