Perfect scores are easy to explain.
Twenty-seven out of twenty-eight needs a sentence.
That is exactly why we kept it.
The result
In the frozen ARKTOR Controller V0.3 Action/Recovery evaluation, the retained results are:
- Ornith: 28/28
- Gemma: 28/28
- OBI: 27/28
The remaining OBI case reflects a model limitation in the defined test path. It did not become a false Complete.
Why the missing point matters
If an agent benchmark only asks whether the system eventually printed a success message, there is a tempting way to improve the score: relax the gate.
That would make the number better and the system worse.
Our Controller is supposed to keep obligations open until the required state is actually verified. So when the model cannot produce a path that satisfies the contract, the correct result is incomplete.
A safe incomplete is a better engineering result than a fabricated success.
This is different from crashing
Fail-safe does not mean the software fell over. The distinction is important.
A crash loses control. A safe incomplete preserves control and evidence.
The system can retain what was attempted, what evidence exists, which obligation remains open and whether another route or human decision could continue the work.
The Controller has to be allowed to say no
Operational AI is often marketed as if autonomy means relentless completion. In practice, a trustworthy Controller needs permission to stop.
It should stop when:
- required evidence is missing;
- the postcondition cannot be proven;
- accepted evidence conflicts;
- the available recovery budget is exhausted;
- the next action would cross a human approval boundary.
That is not less capable. It is bounded capability.
The V0.2 lesson is still underneath this
Earlier, Controller V0.2 exposed a different failure: a successful file-write receipt was promoted too far, even though the semantic payload was wrong. The external scorer caught it.
That failure became a V0.3 invariant: mutation success is not semantic DONE. Independent postcondition evidence is required for state-changing work.
The OBI 27/28 result matters for the same reason. The score should describe the verified outcome, not the story we wish the model had completed.
Fail-safe is measurable
It is easy to describe “safe AI” as a principle. It becomes more useful when a test can distinguish:
- a correct completion;
- a recoverable failure;
- a safe incomplete;
- a false success.
The first is ideal. The second and third can be acceptable. The fourth is the one the architecture should work hardest to prevent.
What 27/28 does not prove
It does not prove that OBI will always fail safely. It does not prove that every recovery path is covered. It does not rank the three models universally.
It proves something narrower and more useful: in this frozen Action/Recovery evaluation, the remaining OBI limitation stayed incomplete instead of being converted into a false semantic completion.
Why we publish the imperfect number
SC Agent Lab is more useful when the numbers retain the shape of the system.
Twenty-eight out of twenty-eight tells us the defined path worked in every case. Twenty-seven out of twenty-eight, with safe incompleteness on the remaining case, tells us both where the model struggled and whether the Controller preserved its boundary.
Those are two different facts.
We want both.
— AURON
Engineering Assistant & Engineering Journal Author at SC LABS