CometAPI with LlamaIndex: example context settings are not a model-capacity contract

General
1

When adapting CometAPI’s LlamaIndex guide for a RAG application, don’t carry context_window=4096 and max_tokens=256 into your configuration as though they were the selected model’s documented limits. They are arguments in an example whose model is still a placeholder.

What the example actually establishes

The guide imports CometLLM from llama_index.llms.cometapi, then constructs it like this:

llm = CometLLM(
    api_key=api_key,
    max_tokens=256,
    context_window=4096,
    model="your-model-id",
)

It follows that setup with llm.chat(...) and llm.complete(...) examples. This shows where the configuration goes. It does not identify a concrete model whose supported context capacity is 4,096 tokens or whose maximum output is 256 tokens.

There is a useful cross-check on the same page: the “Using other models” tip shows CometLLM(api_key=api_key, model="your-model-id", max_tokens=1024). That is another example setting—not evidence of a 1,024-token output ceiling either. The guide points to the CometAPI Models page for available model options, but this integration example is not itself a per-model limits table.

Before copying those values into a RAG configuration

My recommendation is to make the model choice and the budget decision explicit:

  1. Resolve the placeholder first. Record the exact CometAPI model ID and endpoint you intend to use. Obtain documentation for that combination’s context capacity and output limit; don’t fill a missing value with 4096 or 256 from this snippet.
  2. Trace the arguments beyond the constructor. Check how your installed LlamaIndex CometAPI integration uses context_window and max_tokens. Passing a value into CometLLM(...) alone does not establish how it affects prompt construction or what the endpoint enforces.
  3. Budget the assembled request, not just the retrieved passages. Account separately for instructions, the user question, any included conversation history, retrieved context and the requested completion. Apply the selected model’s documented accounting rules rather than treating the example’s two numbers as a universal budget formula.
  4. Test the boundary you plan to rely on. Use representative assembled requests and inspect what happens near the documented limits. If you need context reduction, establish how your application will do it rather than assuming this guide promises automatic truncation.

The decision is not “these numbers are wrong.” They may suit a particular application once checked. The decision is not to promote sample configuration into a model-capacity contract. This guide shows CometAPI integration wiring and simple calls; it does not supply a tested retrieval workload, prove truncation behavior, or verify the limits of the model you will substitute for your-model-id.

2

I would add an economic acceptance test before settling on those application budgets: compare cost per accepted answer, not just tokens per request. A smaller completion budget is not a demonstrated saving if the resulting answer needs another attempt to meet the task’s requirements.

Once the exact model and endpoint limits are documented, I would hold the model, retrieval inputs and evaluation questions fixed, then compare a few supported budget settings. Define acceptance before running the comparison—for example, whether the answer addresses the question and supports its key claims with the supplied passages. Record billed cost across all attempts, accepted-answer count and end-to-end latency including retries. Divide total billed cost by accepted answers; if none pass, report the configuration as failing rather than assigning it an attractive per-request price.

That separates two decisions: whether a configuration fits the documented capacity, and whether it delivers an acceptable answer at an acceptable cost and latency. The LlamaIndex example alone does not answer the second question. This is a proposed evaluation method, not a measured savings claim; neither the sample 256 nor the alternative 1024 establishes an economic optimum.