How to Test AI Agents Across Multiple Models Without Tuning to One Model

Why one-model success can mislead

Agent systems are often developed while one model is used repeatedly. Over time, prompts, parsers and recovery heuristics can quietly adapt to that model’s preferred wording and behaviour.

The result may look like a strong agent while actually being a strong pairing between one model and one implementation.

Multi-model testing asks a harder question: does the controller own the workflow, or is the workflow depending on one model behaving exactly as expected?

Freeze the contract before changing the model

The strongest comparison keeps the system around the model fixed. That includes:

• the user goal;
• required obligations;
• available tools and permissions;
• tool schemas;
• retry budgets;
• evidence rules;
• postcondition checks;
• the semantic definition of Complete.

If those rules change between models, the test no longer tells you which differences came from the model.

Use held-out models, not only development models

A held-out model is useful because it did not shape the controller during development. It can expose assumptions hidden in prompts, parser logic or action sequencing.

The point is not to declare one model universally best. The point is to test whether the controller can preserve the same safety and completion semantics when model behaviour varies.

Measure outcomes at more than one layer

A useful multi-model evaluation separates at least four questions:

• Reasoning: did the model identify a plausible next action?
• Execution: did the permitted tool run?
• Recovery: what happened when an action or assumption failed?
• Completion: did independent evidence prove the required final state?

A model can be excellent at the first two and still be unreliable at the fourth.

Safe incompletion is not the same as failure

For operational systems, one of the most important distinctions is between:

incorrect Complete — the system claims success without sufficient proof;

safe incomplete — the system cannot satisfy the contract and refuses to manufacture certainty.

The second outcome may reduce a benchmark score, but it is often exactly what a bounded controller should do.

Keep model-specific fixes out of the core when possible

If one model needs a special parser, prompt or recovery rule, first ask whether the controller contract can be made clearer generically. A model-specific workaround can be justified, but it should be visible and isolated rather than silently redefining the benchmark.

Example: ARKTOR Controller V0.3

SC LABS used held-out model evaluation as one of several Controller V0.3 gates. On the frozen held-out controller tasks, Ornith reached 20/20, OBI 20/20 and Gemma 18/20.

Action/Recovery evaluation produced Ornith 28/28, Gemma 28/28 and OBI 27/28. The remaining OBI limitation failed safe/incomplete rather than being promoted to a false Complete.

A practical multi-model test plan

1. Freeze the controller version and test corpus.
2. Define acceptable evidence and semantic completion before execution.
3. Run the same task set with the development model.
4. Run at least one held-out model without changing the contract.
5. Record tool success separately from goal success.
6. Record recovery and fail-safe incompletion separately from false success.
7. Inspect model-specific failures for generic controller weaknesses.
8. Only then decide whether a model-specific adapter is justified.

The useful principle

A model is a replaceable reasoning component. If changing the model silently changes what “done” means, the controller does not fully own completion yet.