DO test the whole workflow
Start with a real task and follow it all the way through: model output, tool selection, permission check, execution, result validation and state update. A correct chat response is only one checkpoint.
DON'T call a model answer an agent proof
If the model says the correct thing but the file was never written, the browser never clicked or the API response was never validated, the workflow did not succeed.
DO separate model quality from provider quality
Record the pair: model + provider. In SC LABS testing, the same model produced different scores and different failure modes through different routes. Provider rate limits and upstream errors can dominate apparent model quality.
DON'T trust a catalogue entry
A model appearing in a provider list proves discovery, not callability. Use explicit states such as DISCOVERED → CALLABLE → TESTED → PROVEN, plus DEGRADED, UNAVAILABLE and EOL.
DO validate “successful” responses
HTTP 200 can still contain an unusable payload. Validate schemas, required fields, tool arguments and semantic invariants before changing agent state.
DON'T let requests grant themselves permission
A tool request may ask to write a file or run a process. It must not be able to switch the session from read-only to write-enabled by itself. Authority should come from the user, product policy or a separate approval boundary.
DO make destructive and irreversible actions explicit
Deletion, sending, publishing, purchasing, rebooting and credential changes deserve their own gates. Preview where useful. Log the target. Prefer reversible actions when the product can achieve the same goal.
DON'T hide failure behind unlimited retries
Retries need a reason, a bound and a backoff policy. Repeating a rate-limited or malformed request indefinitely converts one provider problem into an agent reliability problem.
DO preserve the difference between failed and unknown
A timeout after sending a request is not always the same as “nothing happened”. Agents that cannot represent unknown outcomes are prone to duplicate actions.
DON'T use benchmark scores as universal rankings
Benchmark design matters. A 24/24 recognition result and a compiler-verified code-generation result answer different questions. Publish task shape, verifier and limitations.
DO test failure paths deliberately
- provider timeout,
- HTTP 429 or overload,
- empty-success response,
- tool returns non-zero exit,
- permission denied,
- network disconnect,
- stale state,
- missing file or changed UI,
- context exhaustion.
DON'T make every optional capability part of one privileged core
Browser automation, UI control, document parsing and remote transport have different risks and dependencies. Keeping them modular makes the trusted hot path easier to understand and test.
DO keep a fallback for important work
A fallback can be a known local model, another provider, a lower-capability safe mode or a human handoff. The right fallback depends on the task, but “hope the provider comes back” is not a recovery design.
DON'T claim “free forever” without the operational details
State the quota, rate limits, provider, model availability, retry behaviour and expected workload. One free request and a durable multi-step agent are not equivalent products.
DO publish failures
Failures reveal architecture. A 503 teaches you about route resilience. A malformed tool call exposes validation requirements. A permission denial proves whether the boundary actually exists.
A compact review table
| Area | Do | Don't |
|---|---|---|
| Model | Test task-specific capability | Assume size or reputation equals fit |
| Provider | Track health per route | Assume listed means callable |
| Tools | Validate arguments and results | Trust prose confirmation |
| Permissions | Grant externally to the request | Allow self-elevation |
| State | Represent success/fail/unknown | Assume every call is atomic |
| Recovery | Bound retries and provide fallback | Loop forever |
| Evidence | Retain PASS and FAIL | Publish only the best screenshot |
The simplest rule
If you cannot explain what happens when the model, provider, tool or network fails, you do not yet know how the agent behaves.
Test that before calling it production-ready.