1. Which exact model?
Do not accept “frontier model”, “120B model” or “best free model” as enough detail. Record the exact model identifier and version where available. Providers can expose similarly named models with different routing or lifecycle states.
2. Which exact provider route?
The model name alone is not the deployment. The same model can behave differently through different providers because rate limits, shared pools, upstream routing and response formats differ.
3. What does free actually mean?
- requests per day,
- requests per minute,
- token caps,
- concurrency limits,
- model-specific restrictions,
- shared-pool capacity,
- expiry or account conditions.
A multi-step agent can consume several model calls for one user request. Convert the provider quota into real workflows per day, not just API calls.
4. Is the model merely listed or actually callable?
Use a maturity ladder:
DISCOVERED → CALLABLE → TESTED → PROVEN
Add DEGRADED, UNAVAILABLE and EOL. A catalogue page is a discovery source, not a reliability certificate.
5. How many repeated calls were tested?
One successful response proves almost nothing about sustained behaviour. Repeat the same fixed tasks. Record status code, latency, response validity and task result for every call.
6. Does HTTP 200 contain a usable answer?
Validate required response fields. SC LABS testing observed successful HTTP status responses without a usable choices payload. An agent must treat that as a failed model result.
7. Does tool use work?
Chat quality is not enough. Test structured arguments, schema compliance, tool selection, result handling and the agent decision that follows the tool result.
8. What happens at 429, 503 or timeout?
Ask for the recovery policy. Good answers include bounded retry, backoff, circuit breaking and a fallback. “Retry until it works” is not a durable design.
9. Can the route change silently?
Free services may adjust available models or upstream routes. Your agent should be able to detect that a previously proven route is now degraded rather than assuming yesterday's state is permanent.
10. Is there a fallback you have already tested?
A fallback should be proven before the primary route fails. It may be another provider, a local model, a safe reduced-capability mode or a human approval path.
11. Are failures counted in the published score?
A benchmark that excludes provider errors can make a fragile route look excellent. Keep two dimensions: task quality and operational reliability.
12. Is the claim about learning or production?
A free route can be excellent for experimentation even if it is unsuitable for a durable agent. The claim should match the intended workload.
A quick scoring sheet
| Question | Good evidence | Warning sign |
|---|---|---|
| Model | Exact identifier | “Best model” |
| Provider | Named route | Not disclosed |
| Quota | Measured current limit | “Unlimited” without terms |
| Reliability | Repeated fixed run | One screenshot |
| Tools | Schema + execution proof | Chat only |
| Failures | Logged and included | Removed from score |
| Fallback | Already tested | None |
What SC LABS observed
In our Huawei provider lab, ten NIM routes completed a 300-evaluation campaign, but several models were affected by 404, 410, 429, 500, 503, 529 and timeout failure modes. A later OpenRouter A/B showed that the same model or task family could behave materially differently through another route.
That is why our rule is simple: route by proven model-provider pairs and current health, not by a model name in a catalogue.