Freeze the candidate
Evaluation is meaningful only when the code, task suite and success criteria are fixed for the run.
Test more than happy paths
Include incorrect model output, missing evidence, timeouts, unavailable tools and state-changing tasks that require outcome verification.
Use held-out tasks
A candidate that only succeeds on the tasks used during development may be overfit to its own test suite. Held-out scenarios provide a better signal of generality.
Require real integration evidence
Unit tests are necessary but not enough. Run the candidate through the actual product boundaries and physical environments it is expected to coordinate.
Use a release-gate record, not a single pass count
Keep four answers together: which exact build was tested, which failures were deliberately triggered, which tasks were withheld during development, and which real integrations were observed after execution. A result without its environment and acceptance criteria cannot be reproduced.
- Pass: the frozen build meets every required gate within the stated scope, including independent verification of state-changing outcomes.
- Incomplete: a required device, permission boundary or recovery scenario could not be tested. Preserve the open item; do not mark it green.
- Fail: a reproduced result violates a requirement. Fix it, freeze a new candidate and run the relevant checks again.
Even a passing controller candidate is not automatically a public release. Installation, authentication, updates, support and clean-machine acceptance are separate product decisions. See product proof versus release readiness for that boundary.
Publish the evidence
Test counts, environments, failures and limitations make the candidate easier to evaluate.