When bigger isn’t better

The OpenAIs and Anthropics of the world are betting that the more powerful a model is, the more comprehensive, the better it’ll be at solving every kind of problem. Our data’s saying otherwise.

We ran three head-to-head benchmarks against frontier models—one in legal (contract clause detection), one in biomedical research (identifying scientific relationships in published papers), and one in aviation safety (turning NASA incident reports into analyst-like synopses).

We picked these fields because the outcomes of these workflows can’t just be “good enough,” they have to be right. And while not every AI-driven workflow has to be near-perfect, you certainly don’t want to gamble on business or customer-critical tasks.

Trained SLMs get the right answer more often

Frontier models can do a little bit of everything, which is great when you’re planning a trip to Mexico City or trying to get more zinc in your diet. It’s less great for when you need to extract the right answer from thousands of different documents.

Our specialist models (trained on a highly specific task and real data) don’t have that problem.

7x better at quoting a contract clause word for word

We kicked off our analysis by asking 4,000+ questions across 100+ contracts. For each one, the model had to say:

1. Whether a specific clause was in the document
2. Quote the exact wording of that clause

Something a real lawyer (ok, maybe a first-year associate) would do: skim a contract, find a clause, quote it verbatim so that they can brief their boss on risk, cite it in a redline, or flag it for a client.

The Overmind-trained model made roughly half as many mistakes as the frontier flagship model (91.6% versus 82.0%). It was also 7x more accurate at quoting clauses verbatim (59.6% versus 8.5%).

4x more accurate at reading a research paper

Same idea, different domain. This time, we pointed two models (Overmind-tuned Qwen 9B model and a frontier flagship model) at BioRED, a dataset of biomedical research papers human researchers have already annotated.

We hid those annotations from the models, of course, and then asked questions like, “Is there a relationship between chemical A and gene B in this paper?” across 10,000 different entity pairs. The models then had to:

1. Answer yes or no.
2. Return the answer in JSON format, so that, theoretically, that answer could be used in a downstream pipeline.

Again, this is common drudge work in biomedical research when there are no answers yet: pulling results from published papers so a researcher can feed them into a meta-analysis or design a completely new experiment.

The Overmind-trained model got the right answer 4x more often than the frontier flagship (54.9% versus 14.1%). It also returned answers in the required format every single time.

0.7% doesn’t sound terrible, but that’s still 71 findings a researcher has to redo by hand (or that get dropped from an analysis entirely).

10% closer to how a NASA analyst would’ve summarized a safety report

For our third experiment, we ventured into an even more mission-critical domain: aviation safety. NASA’s Aviation Safety Reporting System (ASRS) collects confidential incident reports from pilots, controllers, mechanics, and dispatchers.

We first trained our models on 1,000 such reports. Then, we gave our models and a frontier model 500 fresh reports with an identical prompt. For each report, the models were asked to:

1. Read the whole thing (reporter role, aircraft, flight phase, and the pilot’s account of what happened).
2. Write a single line to go in NASA’s searchable safety database, in the style an analyst would use.

Safety analysts write these synopses so that an investigator working a similar case can find it. Get the synopsis wrong, and the record might as well not be there. To see how well the models did, we compared them to the actual version an analyst wrote across two axes: meaning and wording.

Our 12B model matched the frontier model on meaning—0.657 versus 0.655, statistically a tie—and beat it on wording (0.295 versus 0.268), meaning its synopses used more of the same terms an analyst would’ve picked.

Even the smallest model we trained nearly matched the frontier model on meaning and performed better on wording at, as we’ll see, a considerable difference in cost.

Trained SLMs make up the wrong answer less often

Hallucinations are a liability in any workflow, but especially in ones where the output is used for something very official—a legal filing, a published abstract. In our analysis, frontier models made things up orders of magnitude more often than our Overmind-trained specialist models.

20-30x fewer phantom clauses in legal contracts

If you remember, we asked models to identify a specific clause in a contract and quote its exact wording. Sometimes the clause was in the document, sometimes it wasn’t. Frontier models had particular trouble with this, hallucinating in two ways:

• “False alarms,” i.e., claiming a clause was there when it wasn’t.
• Fabricating entirely new quotes that did not exist.

Per question, the frontier flagship model we tested fabricated quotes from contracts 28x more often than Overmind, and raised false alarms 15x more often.

7x fewer invented findings in biomedical research

In our BioRED test, we counted “hallucinating” as claiming a relationship exists when it doesn’t. Not quite the same as accuracy, since the model can also get an answer wrong by saying, ‘No, there is no relationship’ when, in fact, there is one.

Turns out, frontier models said there was a relationship when there wasn’t a lot. More than seven times more often than the SLM we fine-tuned with Overmind.

That means roughly half of every “yes” the frontier model gave was a relationship that didn’t appear anywhere in the BioRED dataset. Thankfully, this was just a test.

2.5x fewer invented details in aviation safety records

Safety synopses are dense with specifics, all kinds of numbers, identifiers, aircraft, and flight details. So in our ASRS test, we counted how often each model added a specific detail to its synopsis that did not appear anywhere in a report it was given.

The frontier model did this in more than 1 out of every 3 synopses.

The smallest model we trained hallucinated a detail—which could make or break a case—60% less often than the frontier model.

Trained SLMs cost substantially less

Frontier APIs are cheap to rent at first, especially with all the free credits. But as usage grows (a given as you train, use, test, and retrain it over time), your bill grows with it.

Before long, a chunk of your AI budget is paying frontier prices for work a smaller, task-specific model could handle just as well—and in many cases, better.

20x cheaper legal contract review

We broke down how much it costs for each model to answer 1,000 questions.

While the frontier mid-tier model wasn’t so egregious (though double Overmind’s cost), the flagship model was an eye-watering 20.3 times more expensive.

And if you recall, we asked 4,000+ questions. At scale, these models really add up.

35% cheaper biomedical extraction

Overmind processed all questions (over 10,000) for $17.49. On the frontier flagship model, it costs $26.73. That’s a smaller gap than the legal test, but when you’re running a real research pipeline, cutting your bill by a third is significant.

14 to 23% fewer tokens per aviation synopsis

On the aviation test, the frontier model averaged 42.3 words per synopsis. The models we trained averaged between 14 and 23% fewer (32.5 to 39.8 words).

Regardless of how much you pay per token, using fewer of them gets you money back. Plus, you’re using an open-weights model (aka yours to continue fine-tuning) and running it on hardware you already control.

Do you really own your intelligence?

Not if you’re renting it from someone else. And if specialist models offer more control, better performance on specific tasks, and cost less, what’s stopping you?

Often it’s the belief that owning a model is a huge lift. With Overmind, it doesn’t have to be. We’re making it faster, cheaper—technically trivial—to capture production data, fine-tune open-weight models on your own traces, automatically evaluate and improve your agents, and deploy them.