Assume every external step can fail
Providers rate-limit. Networks disappear. Tools return non-zero exits. UI elements move. A remote action can time out after it was already accepted. Reliability starts by representing those outcomes explicitly.
Use more than success and failure
At minimum, distinguish not started, running, succeeded, failed and outcome unknown. Unknown is critical when repeating the action could create a duplicate side effect.
Make important actions idempotent where possible
Reading a file twice is usually harmless. Sending an email, purchasing an item or deleting a record may not be. Give actions stable IDs, check existing state and design retries so they do not blindly repeat irreversible work.
Bound retries
A retry policy should answer: which errors are retryable, how many attempts, what delay, and what happens afterwards? Rate limits deserve backoff. Invalid schemas usually need a different response, not the same request repeated ten times.
Add a circuit breaker
If a provider or model route has a recent failure streak, stop routing new work there temporarily. This protects the agent from turning a provider outage into a queue of doomed retries.
Health-gate model-provider pairs
Track DISCOVERED, CALLABLE, TESTED and PROVEN separately, then DEGRADED, UNAVAILABLE or EOL. Store the provider with the model because the same model can be healthy on one route and unusable on another.
Keep a proven fallback
Fallbacks can include another provider, a known local model, reduced-function safe mode or a human handoff. Test the fallback before you need it.
Checkpoint multi-step work
After each meaningful step, persist enough state to know what is complete and what is next. Do not require the entire conversation history to reconstruct operational truth.
Separate context from durable state
The model's context window is not a database. Context can be truncated, summarised or lost. Keep authoritative workflow state in a format that survives model swaps and context compaction.
Verify side effects
After a write, read back or hash the target where practical. After a build, inspect the exit and tests. After a remote command, validate the returned result. Do not allow the model's narrative to become the source of truth.
Know when to stop automatically
Escalate when an action is high impact, the outcome is unknown, repeated attempts fail, the fallback is also degraded or the requested authority changes. A human approval is a reliability mechanism, not merely a safety inconvenience.
Retain enough audit to debug the failure
Store task ID, model/provider route, tool, permission decision, timestamps, result status and recovery decision. Avoid storing secrets or unnecessary user data in the audit trail.
Test the recovery path deliberately
| Failure | Expected behaviour |
|---|---|
| 429 | Backoff / route health degrades |
| 503 / timeout | Bounded retry then fallback |
| HTTP 200 invalid payload | Reject result, do not mark success |
| Tool non-zero exit | Capture error and stop/repair |
| Outcome unknown | Verify external state before retry |
| Permission denied | Do not self-elevate |
| Context pressure | Compact context without losing durable state |
A minimal durable loop
Plan → Gate → Execute → Verify → Checkpoint → Continue
On failure:
Classify → Retry if bounded → Fallback if proven → Escalate if uncertain.
This is less magical than an “autonomous forever” loop. It is also much easier to reason about when something goes wrong.