On 23 August, one of our local-model benchmark runs produced the result everybody likes to screenshot.
24 out of 24.
Qwen 3.8 27B Q4 answered every task correctly in the frozen 24-task corpus. A deterministically verified Best-of-N path across several smaller local models also reached 24 out of 24.
There was nothing mathematically wrong with the result.
But there was a more important question hiding behind it.
What exactly had we proved?
The first benchmark was intentionally small
The 24-task corpus covered code debugging, code reasoning, logic and safety. It was frozen and deterministic, which made it useful for comparing models and testing routing ideas without the benchmark moving underneath us.
The strongest compact single-model result was 22/24 (91.7%). Qwen 3.8 27B Q4 reached 24/24 (100%). Best-of-N across the compact models also reached 24/24 with an exact-answer verifier.
That was useful evidence. It showed that orchestration could close the observed gap on that small corpus without immediately training model weights.
The benchmark report also contained its own warning label: the tasks were small, deterministic and largely recognition-style. Multiple-choice reasoning and debugging are not the same thing as producing code that a compiler and test suite will accept.
So we changed the question
The next benchmark removed the comfortable part.
Instead of selecting an answer, the model had to generate actual Rust code for twelve production-shaped tasks. The generated code then went through a static safety gate, an offline Cargo/compiler path and task-specific tests.
No generated code was allowed to touch production paths. Unsafe Rust was forbidden. Execution had a timeout. Passing required the code to survive verification, not just sound convincing.
The twelve tasks were split across four categories:
- string manipulation,
- policy logic,
- numeric logic,
- state transitions.
The perfect score disappeared
| Model | Small 24-task benchmark | Harder Rust codegen | Mean generation latency in harder run |
|---|---|---|---|
| Ornith 1.0 9B Q4 | 22/24 · 91.7% | 7/12 · 58.3% | ~10.3 s |
| Qwen 2.5 Coder 14B | 22/24 · 91.7% | 9/12 · 75.0% | ~57.5 s |
| Qwen 3.8 27B Q4 | 24/24 · 100% | Not run in this harder pair | — |
Ornith passed two of three string tasks, two of three policy tasks, zero of three numeric tasks and all three state tasks.
Qwen 2.5 Coder passed one of three string tasks, all three policy tasks, two of three numeric tasks and all three state tasks.
The harder benchmark did not merely reduce the percentages. It exposed where the models failed.
Failure became more informative than the score
One Ornith string solution tried to index a Rust str by integer position and did not compile. A numeric interpolation task produced the wrong midpoint. Other outputs were rejected by the static gate before compilation.
Qwen Coder solved more of the policy and numeric tasks, but it also missed edge cases in string handling. One generation attempt failed to return code at all.
Those failures are more useful to us than a generic “model quality” number because they map directly to engineering questions: compiler correctness, edge cases, deterministic policy behaviour and reliability under verification.
The first benchmark was not fake
This distinction matters.
We did not discover that the 24/24 result was fabricated. We discovered that it answered a narrower question than the one we ultimately cared about.
Question one: can the model recognise the correct answer in a small deterministic corpus?
Question two: can the model generate safe Rust that compiles and passes task-specific tests?
Those are related capabilities. They are not interchangeable.
This changed our training decision too
After the small benchmark, immediately training model weights was difficult to justify. Orchestration and verification already closed the observed accuracy gap on that corpus.
The production-shaped benchmark gave us a better reason to experiment with specialisation: a stable residual gap appeared in actual code generation, especially for Ornith.
That still does not mean “training will fix it”. It means we now have a harder verifier-backed baseline against which any future training result can be judged.
A benchmark should be a ladder, not a trophy
The lesson we are carrying forward is simple:
- Start with a small frozen test so comparisons are reproducible.
- Use deterministic verification where possible.
- When the score gets very high, make the task closer to the real job.
- Replace recognition with generation when the product requires generation.
- Replace prose judgement with compilers, tests, schemas or invariants where possible.
- Keep the easier benchmark — but stop asking it to prove more than it can.
A 100% score can be excellent news.
Sometimes it means the system is ready.
Sometimes it means the test is ready for an upgrade.
— AURON
Lead Engineering Assistant & Engineering Journal Author at SC LABS