A local model can research a task, present an appealing plan and write a large amount of code. The user still needs to know one thing: did the finished result work?
We ran a staged test of a locally served 27B variant that had to move through research, planning, coding and browser review. We assessed the actual result against fifteen checks rather than relying on how confidently the model described its progress.
It failed the autonomous-completion gate.
The broader ARKTOR product direction keeps model output, computer control and verification in separate roles so a model cannot simply declare an unverified result complete.
What the scorecard recorded
The run passed ten of fifteen objective checks and achieved 66.7% on that specific scorecard. The recovered output could be made renderable, but the model had not completed the task autonomously.
The recorded bottleneck was unambiguous: all six model-generation attempts ended because the generation length limit was reached. The resulting HTML, CSS and JavaScript were truncated. JavaScript syntax checking failed before recovery.
The machine did produce tokens at a respectable local speed. The measured rate did not rescue the result. This is precisely why tokens per second cannot stand in for completed work.
The sanitised scorecard makes the outcome checkable: 10/15 objective checks, no autonomous completion and six generation-length failures.
What to test in an agent survival challenge
A reproducible test should include:
- A fixed brief with clear deliverables and constraints.
- Separate, safe workspaces for each model run.
- A frozen tool interface and runtime budget.
- Exact acceptance checks for files, scripts, layout and required content.
- A real browser or independent renderer.
- Failure handling and bounded correction attempts.
- A final result that distinguishes autonomous pass, recovered result and failure.
Output limits deserve special treatment. The right recovery is not to assume the truncated file is nearly finished. The runner should detect the length termination, preserve valid progress and continue or re-plan within explicitly allowed limits.
Model selection also needs interaction-fit tests rather than one global prompt.
The read-back guide describes the checks that can keep incomplete files from being mistaken for a finished task.
Don't confuse recovery with a clean pass
If a human or a supervising system repairs invalid JavaScript, that may demonstrate a valuable workflow. It does not mean the original model completed the job unaided.
Report at least three separate outcomes: original autonomous result, independently checked artefact and any recovered result after intervention. A scoring system that merges them rewards invisible help.
What a better next run would look like
Break the project into bounded files or stages. Validate the output after each stage. Make the model respond with a manifest of intended files and verify that every promised file exists and parses.
Only when the code gate passes should browser QA begin. Re-run the same challenge across models with a consistent budget and publish the failures as well as the successes.
The SC LABS agent testing method describes the wider verification chain, and our landing-page benchmark guide gives a smaller practical starting point.
Evidence boundary: This write-up concerns one retained October 2026 test of one configured model and its runner. It cannot rank all 27B models or establish broad product readiness.