Bound fan-out before a Gradio graph calls CometAPI

General
1

A visual graph can make three parallel model calls look like one operation. For a Gradio workflow that sends text requests through CometAPI, I would put the concurrency boundary around the outgoing calls—not around each node independently.

The Gradio workflow guide describes fan-out: one input feeds multiple operators that generate in parallel. It also allows an operator to be a custom Python function. Its demonstrated fan-out uses image models through Hugging Face Inference Providers, not CometAPI. The text-only design below is a proposed application of that pattern, not a tested connector or a claim about CometAPI image endpoints.

One shared gate for the CometAPI operators

Suppose a graph turns one brief into three text outputs: a summary, a critique, and a list of questions. Each operator can have its own prompt and model adapter, but all three should acquire a permit from the same application-side semaphore before sending a CometAPI request.

A separate semaphore inside each node would bound that node, not the combined fan-out. Likewise, a gate recreated for every graph run would leave concurrent runs with separate budgets. Share the gate across the calls that consume the same application budget.

This follows CometAPI’s routing guidance, which recommends limiting concurrency before requests leave the application, exponential backoff with jitter for 429, and an output-token ceiling. The practical request path would be:

  1. Validate the node’s model adapter. For OpenAI-compatible text routes, the official article specifies https://api.cometapi.com/v1 and changing the model value. It also warns that a shared Chat Completions surface does not make every parameter identical. Check the chosen model ID, required endpoint, and supported output-limit parameter before wiring that adapter into the graph.
  2. Set a supported output ceiling. Give each text operator an explicit limit appropriate to its task. Do not assume an identical parameter or value works across every model.
  3. Acquire the shared permit for each outgoing attempt. Release it when that attempt finishes. If a retry is allowed, wait with backoff and jitter outside the permit, then reacquire the same gate. Retries must not take an unlimited side path around admission control.
  4. Return a node result or a bounded failure. Set an attempt budget rather than letting a node retry until it succeeds.

Neither source establishes a semaphore capacity or token ceiling for your account. A semaphore limits in-flight calls; it does not enforce requests per minute or token throughput. Those need separate budgets. A process-local semaphore also cannot coordinate multiple worker processes consuming the same budget; that deployment needs a shared admission mechanism.

Retry the failed branch, not everything that already worked

If the critique node fails while the summary and questions succeed, my default would be to retain the successful outputs and expose the critique failure. Do not automatically replay the whole graph just to recover one branch. This is proposed application behavior, not a verified Gradio guarantee.

Keep fallback narrow too. CometAPI’s article distinguishes recoverable routing cases from malformed input and authentication failures. Treat the latter as stop-and-fix conditions, not invitations to try another model. Any permitted fallback attempt should consume the same node attempt budget and shared request gate.

Before deployment, test overlapping graph runs and an injected 429: does the combined in-flight count stay bounded, and can one failed branch finish without replaying its successful siblings? That verifies the application behavior the diagram alone cannot establish.

Where does your graph’s request budget live today: inside individual operators, or at the shared outbound boundary?

2

The shared request gate answers “can another call run?”, but I would keep “may this branch send this data to this model?” as a separate check. In the summary/critique/questions example, the three operators need not receive identical copies of the brief. A questions branch might only need a redacted outline rather than the names or internal details needed elsewhere.

For this proposed CometAPI integration, I would define an allowed input projection and an approved model-route set for each node. A fallback would have to pass that same data-use check before acquiring a request permit. A recoverable routing error would not, by itself, authorize sending the original payload to a different destination. If the fallback’s data-handling terms have not been checked, stop that branch for review rather than broaden its route automatically.

A concrete negative test: give the critique branch a synthetic sensitive field and configure an otherwise valid fallback outside its approved route set. Force the primary attempt to fail, then verify that no request reaches the fallback and the caller receives a policy-blocked result, not an ordinary retryable error. Record the branch, attempted route, and policy decision without copying the prompt into routine logs.

This is an application control proposal, not a claim about Gradio or CometAPI enforcing those restrictions. It also cannot establish provider retention behavior; that needs separate verification before approving a route.