A local inference optimization is a reason to rerun your routing benchmark—not yet a reason to cut your hosted API budget. The interesting detail in llama.cpp b10844 is the bottleneck it targets: DeepSeek-V4 hyper-connection operations on Vulkan.
What the release actually says
The release adds fused DSV4_HC_COMB/PRE/POST operations. Its notes report that, for DeepSeek-V4-Flash on gfx1151 (Strix Halo), the unfused Sinkhorn comb chain alone takes about 32% of decode op time, spread over roughly 16,000 dispatches per token. That is an upstream observation about the old path, not a benchmark performed here.
The comb kernel runs the full 20-iteration Sinkhorn in registers, replacing about 137 strictly ordered node executions per site with one dispatch. The pre/post kernels handle stream collapse and fan-out, staging per-token coefficients in shared memory.
That mechanism makes the release worth testing if this is your local inference path. It does not establish a 32% end-to-end speedup, a 32% cost reduction, or a break-even point against hosted inference. Decode op time is not the same denominator as a complete API request under load.
One testing trap: the notes first describe per-op environment-variable toggles, then explicitly say the review fixes dropped per-op disables and device flags. Do not copy the earlier per-op toggle recipe as though it describes the final release.
A budget decision needs a workload comparison
My recommendation is to compare a pinned pre-change build with b10844 before changing capacity or routing policy. Keep the GPU, driver, model artifact, quantization, context length, output length and concurrency fixed. Record the exact configuration with the results.
For each representative workload slice, collect:
- Time to first token and decode throughput separately. Do not substitute the latter for the former.
- Complete-request p50 and p95 at expected load, including queueing, plus memory pressure.
- Accepted outputs under the same quality rubric, not just completed generations.
Then compare local and hosted routes over the same evaluation period and request mix. For the local route, include allocated hardware cost, electricity and operations effort; for the hosted route, include API charges, retries and fallbacks. Divide each route’s total by its accepted outputs. If the routes use different model variants or quantizations, treat quality equivalence as something to test, not assume.
This is a proposed evaluation protocol, not a savings forecast. A faster kernel is useful evidence about one component; the budget decision depends on whether the resulting service meets your latency and quality targets at the load you need.
Move only the workload slices that pass those targets and improve cost per accepted output. Until those measurements exist, keep the hosted allocation unchanged rather than treating a pre-fusion bottleneck percentage as money already saved.