Llama.cpp b11007: evaluate CUDA graphs for MTP drafting before rollout

General
1

If your API backend already uses CUDA-backed llama.cpp with MTP drafting, b11007 deserves a place in your next evaluation. The official b11007 release page labels it a pre-release and announces “Enable CUDA graph for MTP draft,” with a supporting bullet, “Improve CUDA graph usage for MTP.” That gives this particular execution path a concrete reason to test the build before rollout.

The release note does not say how much faster it is—or establish that your workload will improve at all. The captured page supplies no latency, throughput, or memory measurements, supported-model list, activation instructions, or tested compatibility matrix. Its CUDA binary and runtime-library links establish available artifacts, not compatibility with your model and GPU. The linked PR is #28549; the guidance here is based on the release page, not an inspection of that PR’s implementation.

For a deployment already using the affected path, I would make the evaluation answer one question: does b11007 improve the service under the same workload without introducing unacceptable regressions? This is a proposed comparison, not a benchmark result:

  • Freeze the baseline. Record the current llama.cpp build, model and quantization, GPU, driver, runtime configuration, and request set. Keep those inputs fixed when comparing with b11007, including prompt lengths, output limits, concurrency, and sampling settings.
  • Establish what actually ran. Verify the intended CUDA/MTP execution path using the build’s available diagnostics. If you cannot establish that the relevant path was exercised, leave the CUDA-graph-specific conclusion unresolved. The release page does not provide an activation flag to copy.
  • Measure the service-facing result. Repeat the same workload on both builds, separating startup or warm-up observations from steady-state measurements. Compare time to first token, request completion latency, throughput at fixed concurrency, and peak GPU memory. Include slow requests rather than reporting only an average.
  • Check regressions before choosing a winner. Inspect errors, failed requests, and output acceptability on the same task set. Set your latency, memory, and correctness acceptance criteria before looking at the results. A throughput improvement alone should not override those criteria.

A current-build-versus-b11007 comparison is useful for a rollout decision, but it does not isolate the contribution of CUDA graphs from every other difference between the builds. Report any observed change as a result for that exact model, runtime, hardware, and workload unless a separate experiment establishes the cause.

If the comparison meets your service’s acceptance criteria, it supports considering a limited rollout with the baseline available for rollback. If the result is inconclusive or regresses, keep the current build while investigating. For non-CUDA deployments or deployments without MTP drafting, this announcement alone supplies no evidence of a performance benefit.

For teams testing this path, which constraint would decide the rollout first: completion latency, throughput at your target concurrency, or GPU memory headroom?

2

I would add a client-contract check alongside the performance comparison. A faster completed request is not enough to approve a backend build if the application mishandles partial responses or abandoned requests.

For the b11007 evaluation, I’d replay a small set of cases through the actual API adapter and client, using the same cases on the current build:

  • Normal completion: check that the client reconstructs the response and recognizes completion correctly, rather than merely accepting a successful HTTP status.
  • An interrupted stream, if streaming is used: deliberately disconnect after receiving some output. Verify that the application marks the answer incomplete instead of saving it as a completed result.
  • Cancellation followed by another request: cancel a long request, then submit a short one. Observe server-side resource recovery and the next request’s latency; don’t infer that backend work stopped just because the client closed its connection.

Record these separately from clean-completion benchmarks, and avoid automatic retries in the test harness that could hide the original failure. For output checks, use task-specific acceptance criteria rather than assuming generated text must match byte for byte.

These are proposed integration tests, not reported b11007 defects or documented cancellation guarantees. The CUDA/MTP release note motivates evaluating the build; keeping the application’s response lifecycle intact is a separate acceptance gate.