DeepSeek-V4 Vulkan fusion: measure before shifting your hosted API budget

A local inference optimization is a reason to rerun your routing benchmark—not yet a reason to cut your hosted API budget. The interesting detail in llama.cpp b10844 is the bottleneck it targets: DeepSeek-V4 hyper-connection operations on Vulkan.

What the release actually says

The release adds fused DSV4_HC_COMB/PRE/POST operations. Its notes report that, for DeepSeek-V4-Flash on gfx1151 (Strix Halo), the unfused Sinkhorn comb chain alone takes about 32% of decode op time, spread over roughly 16,000 dispatches per token. That is an upstream observation about the old path, not a benchmark performed here.

The comb kernel runs the full 20-iteration Sinkhorn in registers, replacing about 137 strictly ordered node executions per site with one dispatch. The pre/post kernels handle stream collapse and fan-out, staging per-token coefficients in shared memory.

That mechanism makes the release worth testing if this is your local inference path. It does not establish a 32% end-to-end speedup, a 32% cost reduction, or a break-even point against hosted inference. Decode op time is not the same denominator as a complete API request under load.

One testing trap: the notes first describe per-op environment-variable toggles, then explicitly say the review fixes dropped per-op disables and device flags. Do not copy the earlier per-op toggle recipe as though it describes the final release.

A budget decision needs a workload comparison

My recommendation is to compare a pinned pre-change build with b10844 before changing capacity or routing policy. Keep the GPU, driver, model artifact, quantization, context length, output length and concurrency fixed. Record the exact configuration with the results.

For each representative workload slice, collect:

  • Time to first token and decode throughput separately. Do not substitute the latter for the former.
  • Complete-request p50 and p95 at expected load, including queueing, plus memory pressure.
  • Accepted outputs under the same quality rubric, not just completed generations.

Then compare local and hosted routes over the same evaluation period and request mix. For the local route, include allocated hardware cost, electricity and operations effort; for the hosted route, include API charges, retries and fallbacks. Divide each route’s total by its accepted outputs. If the routes use different model variants or quantizations, treat quality equivalence as something to test, not assume.

This is a proposed evaluation protocol, not a savings forecast. A faster kernel is useful evidence about one component; the budget decision depends on whether the resulting service meets your latency and quality targets at the load you need.

Move only the workload slices that pass those targets and improve cost per accepted output. Until those measurements exist, keep the hosted allocation unchanged rather than treating a pre-fusion bottleneck percentage as money already saved.

I would separate two questions before using this comparison to change routing: did the backend change preserve output quality, and does the local route meet the same acceptance standard as the hosted route? A local-versus-hosted comparison alone cannot isolate the fusion change if the routes also differ in model or quantization.

For the first question, I would use a paired evaluation: the same prompts, model artifact, quantization and decoding settings on the pinned pre-change and b10844 builds. Keep a held-out prompt set, score outputs without exposing the build labels, and inspect disagreements rather than reporting only the aggregate acceptance rate. If the builds include other changes, label the result a build-to-build comparison—not a fusion-only effect.

I would also define a maximum tolerable quality drop before running the evaluation, separately for consequential workload slices such as structured-output validity and answer correctness. Report the paired acceptance-rate difference with an uncertainty interval. “No statistically detectable difference” is not enough to establish equivalence if the interval still permits a practically unacceptable regression; that slice needs more evidence before migration.

Only after that gate would I compare the local and hosted routes on cost per accepted output. This adds a stop condition to the budget test: a cheaper aggregate result should not conceal a quality regression in a smaller, important request class. This is an evaluation proposal, not evidence that b10844 changes output quality.