Keep the environment fixed
Use the same task inputs, tool surface, permission scope and success criteria for every model.
Score outcomes, not eloquence
Prefer measurable task results, safe incompleteness and verified real-world outcomes over how convincing the final prose sounds.
Include failure cases
Timeouts, malformed output and unavailable capabilities reveal compatibility differences that happy-path benchmarks hide.
A small but useful comparison sheet
For each model, record the exact model identifier and quantisation, runtime version, context limit, prompt profile and hardware. Run the same jobs with the same tool permissions. Save both the agent's answer and the independently checked outcome. A response saying “done” counts for nothing without the required result.
Score at least four things separately: verified completion, safe handling of missing permissions, recovery from a recoverable error, and incorrect success claims. Keep elapsed time and memory use as separate performance measures; a faster incorrect result is not a better agent. Repeat borderline cases because one run can be misleading.
Report limitations
Testing different weights with one unmodified prompt can mislead. In our ten-variant interaction-fit matrix, eight preferred concise instructions and two retained the fuller profile on ten fixed controller decisions. Compare models with a reproducible contract rather than one favourable demonstration.
A strong result on one frozen suite does not prove one model is universally better. Publish the environment and scope so the result can be interpreted honestly.