Direct answerA fail-safe AI agent should only report success when required postconditions are verified. If the evidence remains incomplete, contradictory or exhausted after bounded recovery, the correct terminal state is explicit incompleteness — not a guessed or model-declared DONE.

Failure and false success are not equally bad

Traditional software often treats failure as the thing to eliminate. In an agent system, that is only half the problem. The more serious failure can be a convincing success message that is not supported by the resulting state.

A failed request is visible. A false success can silently propagate into the next step, another system or a human decision.

Give the runtime more than PASS and FAIL

A binary success model is too small for real operational work. Useful states include:

  • VERIFIED / PASS: the required condition is proven.
  • FAIL: accepted evidence proves the attempted outcome is wrong or the operation failed.
  • UNKNOWN / INCOMPLETE: the required condition has not been proven either way.
  • CONFLICT: accepted evidence disagrees.

UNKNOWN is not a weakness. It is the correct representation of missing proof.

Do not let the model collapse uncertainty

Language models are good at producing complete-sounding answers. That is useful for communication and risky for state management. The model should be able to propose the next action, but it should not be allowed to convert uncertainty into verified completion by wording alone.

The gate should live outside the model response: required obligations remain open until software receives acceptable evidence.

Recovery should be bounded

Fail-safe does not mean stop at the first problem. The system can retry or choose another route, but recovery needs a budget.

A practical pattern is:

attempt → observe → verify → recover if useful → stop when verified or when the recovery budget is exhausted.

The budget can limit repeated tool calls, repeated plans, time, cost or risk. Without a budget, an agent can loop forever. Without a safe terminal state, it can instead invent success just to stop.

Incomplete should be actionable

“Could not complete” is not enough. A useful incomplete result should preserve:

  • which obligation remains open;
  • what was attempted;
  • what evidence was obtained;
  • why that evidence did not satisfy the gate;
  • whether another route exists;
  • whether human clarification or approval could unblock it.

This turns incompleteness into a resumable state rather than a dead end.

Postconditions are the anchor

State-changing tasks are where false success becomes easiest to create. A write, click, process start or remote command may all return a successful receipt while the intended state remains wrong.

The safer pattern is:

mutation → receipt → independent observation → postcondition verification.

If the observation cannot prove the requested result, semantic completion stays incomplete.

Fail-safe behaviour should be tested on purpose

A benchmark that only rewards perfect completion can hide the quality of a safe failure. Test cases should therefore include situations where:

  • the model chooses an invalid action;
  • the tool refuses the action;
  • the mutation succeeds but the postcondition is wrong;
  • the evidence source is unavailable;
  • two observations conflict;
  • the recovery budget is exhausted.

The desired outcome is not always completion. Sometimes the desired outcome is no false Complete.

ARKTOR case study

In the current ARKTOR Controller V0.3 Action/Recovery evaluation, Ornith and Gemma reached 28/28. OBI reached 27/28. The remaining model limitation did not produce a false Complete; the workflow remained safely incomplete.

That is why the public result is not rounded up. The missing point is part of the evidence, and the fail-safe behaviour is itself a result worth preserving.

Scope: this is one tested controller evaluation, not a claim that every failure mode or every model will always fail safely. The current published ARKTOR evidence is limited to the defined test suites and retained full-stack workflow.

A practical checklist

  • Can required state remain UNKNOWN?
  • Can contradictory evidence remain CONFLICT?
  • Does a tool receipt remain separate from semantic success?
  • Are recovery attempts bounded?
  • Can the system stop without fabricating DONE?
  • Does an incomplete result preserve enough evidence to resume?
  • Do benchmarks reward safe incompleteness instead of only raw completion rate?

The useful principle

A reliable agent is not one that always says yes. It is one that knows the difference between a proven result, a recoverable failure and an outcome it cannot honestly verify.