A fallback test can pass while telling you nothing about a provider’s uptime. The useful question for a synthetic suite is narrower: given these failure signals, did the application make exactly the calls its policy allows—and stop when it should?
The policy-test section of this error-handling article explicitly separates mocked signals from production reliability claims. It recommends injecting status and error-body signals in staging, checking call count and order, and preserving original request context in the final error. That is a useful boundary for any AI API client with retry and fallback logic.
Build a trace, not just a success assertion
Here is a proposed test setup, not a measured result or a production retry recommendation:
- Configure two fake routes, A then B, with at most two attempts per route, including the first call. Four calls is the ceiling, not a target.
- Replace the transport with scripted responses. Disable hidden retry layers, or explicitly include them in the call budget.
- Inject a fake clock and deterministic jitter input. Record scheduled waits without sleeping through real backoff.
- Record route, attempt, status, normalized error code, synthetic request ID, elapsed time, and decision. Do not capture credentials or full request bodies.
Use A1 → A2 → B1 to mean three actual transport calls. A test that checks only “eventually returned text” can miss extra calls or the wrong route order.
A small acceptance matrix
The error classifications below follow the linked article’s policy. Adapt the classifier to your API’s documented semantics rather than treating HTTP status alone as universal truth. Assume sufficient deadline remains unless a row says otherwise.
| Injected sequence | Expected trace | Assertion that matters |
|---|---|---|
| A returns 401 | A1, then fail |
No retry, no B call; surface authentication failure. |
| A returns a path-related 404 | A1, then fail |
Fallback must not conceal a broken endpoint. |
A returns 500 with error.code: invalid_request |
A1, then fail |
Error-body classification overrides the transient-status rule. Repeat for invalid_request_error. |
| A returns 429, then succeeds | A1 → A2, then return |
A configured backoff occurs before A2; B is never called. |
| A returns transient 503 twice; B succeeds | A1 → A2 → B1, then return |
A’s budget is exhausted before switching; no B2 after success. |
| Both routes return transient 504 throughout | A1 → A2 → B1 → B2, then fail |
No fifth call and no unconfigured third route. |
| Deadline expires during the scheduled wait after A1 | A1, then fail |
Neither A2 nor B1 starts after the shared deadline. |
For retry rows, assert the scheduled delay against your configured backoff and jitter policy—not merely that a sleep function was called. Specify route-transition delay separately so the harness does not accidentally invent one.
The path-related 404 qualifier matters: a separately verified model-unavailable signal is a different fixture, not permission to switch models on every 404.
Make the final failure inspectable
On the all-routes-exhausted case, assert that the caller receives a bounded, sanitized account of the failure: the initial request identifier, attempted routes, relevant error classifications, and the reason execution stopped. A generic “no route completed” exception alone does not prove original context survived. The article asks for context preservation, but its illustrative terminal exception is not evidence that your implementation achieves it.
Add a replay guard fixture as well: if partial output or a tool side effect has already occurred, require controlled failure or an explicitly designed checkpoint recovery path instead of blindly replaying the whole request on B. Keep this separate from ordinary pre-output transient failures.
What a passing report can say
A defensible report is: “For these injected cases and this configured budget, call order, delays, stop conditions, and error-context assertions passed.” This post proposes the suite; it does not report an executed run.
It cannot establish provider availability, independence of route failures, live recovery latency, or equivalent model behavior. Keep live-service observation and output/tool compatibility evaluation as separate gates. A deterministic fallback suite is valuable because it can reject a broken controller before live traffic—not because a fake 503 proves the backup will be there.