Before adding a fallback model to a CrewAI workflow, decide which layer owns retries. Otherwise, “try the next route” may happen only after the current client has already sent several requests—and rerunning the crew can repeat successful work as well.
The practical decision is not how many fallback models to list. It is one attempt budget, one elapsed-time budget, and one policy owner.
The configuration trap
CrewAI’s versioned LLM documentation shows timeout=60.0 and max_retries=3 in an advanced configuration example. That establishes a configuration surface, not a universal default or proof that every provider path implements it identically.
The CometAPI CrewAI tutorial takes the opposite configuration choice in its helper: max_retries=0. Its explanation is explicit: automatic client retries can produce hidden requests before application fallback executes. The transferable lesson is to make application routing and client retries agree, rather than configuring each independently.
Here is the multiplication risk as a hypothetical, not a measured CrewAI result:
- An application permits three total attempts for one logical LLM call, including fallback.
- Each client invocation permits an initial request plus three additional retries.
- If every invocation exhausts that allowance, the call reaches
3 × (1 + 3) = 12transport attempts.
That is not a bound for an entire crew: agents may make other calls, and tools add separate work. Confirm whether your installed client’s setting counts retries or total attempts before using this arithmetic. A per-request timeout is also not a workflow deadline; sequential calls and backoff must fit inside the same elapsed-time budget.
A retry-owner runbook
My recommendation for application-managed fallback is:
| Boundary | Policy to make explicit | Stop condition |
|---|---|---|
| LLM client | Disable automatic retries on the tested path when the application owns them; verify the resulting request count | Observed attempts exceed the configured allowance |
| Application router | Count the initial call, same-model retries, and fallback calls against one cap | No attempts remain |
| Workflow deadline | Include requests, backoff, and recovery overhead; do not start another call without sufficient remaining time | Deadline or cancellation reached |
| Request contract | Validate required tools, output shape, and parameters on each fallback route | No compatible fallback exists |
| External effects | Check durable task/operation identifiers before replay | Prior effect is unresolved or replay is unsafe |
Treat max_retries=0 as a configuration decision to test, not proof that every SDK, proxy, or wrapper has stopped retrying. If a lower layer must retain retries, include those attempts in the budget instead of granting it an independent allowance.
The tutorial distinguishes transient failures—timeouts, rate limits, and temporary server failures—from authentication and malformed-request problems. Use bounded retry or fallback only for the former. For a rate limit, back off within the remaining deadline; switching models should not be assumed to bypass the limit. Authentication, invalid model IDs, unsupported parameters, or malformed requests should stop the loop and route to configuration repair, not consume every fallback route.
Retrying a call is not recovering a crew
The tutorial’s simple fallback loop reruns kickoff(). It explicitly does not claim to identify the failed agent. If research and analysis completed before the writer failed, a whole-crew restart can repeat both earlier stages.
For small stateless work, that trade-off may be acceptable. For expensive tasks or tools that write to external systems, plan checkpoint-based recovery before adding fallback. Verify the checkpoint API against the installed CrewAI version. A restored checkpoint should not be treated as proof that an external write happened exactly once: record operation identifiers and resolve ambiguous effects before replaying them.
What to verify before enabling it
Use controlled failure injection rather than waiting for an incident:
- Force a timeout or temporary server failure. Count actual outbound attempts, including those below the router, and compare them with the shared cap.
- Force an authentication or malformed-request failure. Confirm that the router stops without walking the fallback list.
- Exhaust the elapsed-time budget during backoff. Confirm that another route is not launched and test cancellation of in-flight work.
- Fail a later task after an earlier external effect completes. Confirm that recovery does not blindly repeat that effect.
Record workflow/task identifiers, route, outer attempt number, observed transport attempts, elapsed time, remaining budget, failure class, and stop reason—without secrets or prompt bodies. These are proposed acceptance checks, not reported test results or availability guarantees.
The useful success signal is not merely “fallback returned text.” It is that recovery stayed within the declared budgets, preserved the required request contract, and did not replay work whose outcome was still unknown.