Suppose one logical generation request fails three times and 200 clients all retry on the same fixed schedule:
attempt 1 fails -> every client waits 200 ms
attempt 2 fails -> every client waits 400 ms
attempt 3 fails -> every client waits 800 ms
The delays grow, but the clients still return in waves. Exponential backoff has reduced the frequency of attempts without removing their synchronization.
AWS’s backoff-and-jitter analysis demonstrates this problem with a simulated optimistic-concurrency workload. Its graphs show clustered calls under backoff without jitter. Adding randomness spreads the attempts and reduces contention in that simulation.
That result answers when an eligible retry should run. It does not answer the earlier question: should this request run again at all? A CometAPI client needs both decisions, in that order.
Step 1: classify the outcome
Give the request controller a small closed set of branches:
| Classification | Controller action |
|---|---|
| Success that passes the application acceptance check | Return it; do not retry |
| Request or authentication error that needs correction | Stop and surface the error |
| Temporary same-route failure that the endpoint contract permits retrying | Schedule a bounded retry |
| Primary route is unavailable or unsuitable, and a qualified alternate model exists | Consider fallback within the remaining request budget |
| Outcome is unknown or the surrounding workflow might already have committed a durable effect | Reconcile before replaying the outer workflow |
The exact mapping from status, timeout, or response body to these rows belongs to the application’s verified endpoint contract. A timeout by itself is not proof that no work occurred, and this article does not claim that an undocumented CometAPI request is safe to replay.
Step 2: separate retry from fallback
CometAPI’s official multi-model fallback playbook supplies the product-specific distinction. A retry sends another attempt to the same model or provider for a temporary failure. A fallback changes model or provider when the first route is unavailable or unsuitable.
That difference belongs in code and telemetry:
logical_request_id: case-417
attempt: 2
branch: same_route_retry | model_fallback
route: <verified model route>
reason: <classified failure>
remaining_attempts: 1
remaining_time_ms: 2400
Without the branch field, a model change can be mistaken for an ordinary transport retry. That hides a material behavior change: the alternate model may have different latency, output, tool, or quality characteristics. A fallback should therefore use only a model already qualified for this task, not any route that happens to return a response.
Step 3: jitter only eligible same-route retries
For an eligible retry, Full Jitter can be expressed as:
maximum_delay = min(cap, base_delay * 2^retry_number)
sleep_for = random_between(0, maximum_delay)
The important word is maximum. Each retry samples anywhere from zero to the current capped exponential delay. Clients no longer share one deterministic wake-up boundary.
A minimal controller keeps the eligibility decision outside the delay function:
result = send(primary_route)
decision = classify(result)
while decision == SAME_ROUTE_RETRY and budget.allows_attempt():
wait(full_jitter(retry_number, base_delay, cap))
result = send(primary_route)
decision = classify(result)
if decision == QUALIFIED_FALLBACK and budget.allows_attempt():
result = send(qualified_fallback_route)
return accept_or_controlled_failure(result)
This ordering prevents a backoff library from silently deciding that every failure deserves another call. It also keeps a route change out of the same-route retry loop, where it would otherwise look like one more delayed transport attempt.
Step 4: test the classifier boundary
Build one fixture for every classification row and assert which component is allowed to run:
- a successful accepted response never calls the delay function;
- a correctable request error never calls retry or fallback;
- an eligible same-route failure calls the Full Jitter function before another attempt;
- a qualified route failure emits
model_fallbackrather thansame_route_retry; and - an unknown outer-workflow outcome stops for reconciliation.
Record the classifier result, branch, sampled delay, route, and terminal outcome for each fixture. A separate load test can determine whether the chosen jitter parameters smooth arrivals for the real workload; do not copy AWS’s simulated numerical gains into a production forecast.
The practical rule is simple: classification authorizes an attempt, the shared request budget limits it, and jitter schedules it. In a CometAPI integration, keeping same-route retry and model fallback as separate branches makes that rule observable and prevents a random delay from becoming an accidental retry policy.