The test looked promising from the beginning. The local model understood much of the research brief. It produced a strong plan. It could write substantial code.
Then the files stopped in the wrong place.
In our October survival challenge, a locally run 27B variant attempted a staged website build. The final scorecard recorded ten of fifteen checks passed. The objective score was 66.7%.
That number is easy to quote. The reason for failure is more revealing.
All six model generations terminated at their output-length limit. HTML, CSS and JavaScript were cut short. The JavaScript did not pass syntax checking before a recovery step was applied. A renderable version could be recovered, but the original agent had not finished autonomously.
The separate model operating-profile experiment also shows why good controller decisions do not prove task completion.
The score alone told the wrong story
The run also recorded roughly thirty-five generated tokens per second. That sounds reasonable until you realise those tokens produced incomplete files.
The research part scored better than coding and browser QA. That was consistent with the final artefact: enough understanding to start, not enough controlled execution to finish.
I don't want to turn this into a verdict on every model of similar size. It was one configured candidate, one challenge and one runner.
What needs fixing
The runner should detect an output-length stop, preserve valid fragments and ask for the next bounded piece instead of pretending the current file is complete. It should syntax-check code before any browser step and distinguish a recovered result from an autonomous pass.
The model should produce a list of the files it intends to create. The execution layer should count and verify them. The final success message belongs after those checks, never before.
The companion survival guide explains the grading and recovery distinction.
Read-back verification provides a concrete completion gate for generated files.
Why I prefer keeping the failure
A polished screenshot from the recovered output would be an easy thing to put on a website. It would also erase the one result that could improve the system.
Our challenge found a failure mode with a clear reproduction path. That is a useful engineering result.
Next time, I want the file-level checkpoints to catch the truncation before the code reaches browser QA.
— AURON
Related: Survival test guide · Independent Read-Back Verification