Llama.cpp b11017: Target Vulkan MoE evaluation at the coopmat1 path

General
1

For engineers serving a local MoE workload with llama.cpp on Vulkan, b11017 is a candidate for a targeted evaluation. The first question is whether the workload actually exercises mul_mm coopmat1; the release note gives a reason to investigate that path, but no measured gain to use in an upgrade decision.

The official b11017 release page describes the change as “vulkan: skip unneeded MoE work in mul_mm coopmat1 path,” linking PR #25483 and commit 7490357. The captured page marks the build Pre-release. It supplies no benchmark results, tested model or quantization, hardware test conditions, or correctness results.

That leaves two separate things to establish: whether this change applies to your workload, and whether the new build improves the behavior you care about.

The page lists Vulkan binaries for Ubuntu x64, Ubuntu arm64 and Windows x64. Choosing one of those binaries does not establish that a particular GPU and workload use mul_mm coopmat1. The release page does not explain path selection, so confirming that is a prerequisite for attributing any result to this change. If you cannot establish the execution path, label it unknown in your comparison.

My suggested evaluation would be:

  1. Record the baseline and candidate precisely. Keep the same model file and quantization, GPU, driver, build configuration, runtime settings and measurement harness. Record both llama.cpp revisions. A comparison against an older deployment evaluates the upgrade as a whole; intervening changes prevent attributing every difference to #25483.
  2. Replay a fixed, representative workload. Hold prompts, context lengths, output limits, sampling settings and concurrency constant. Repeat latency measurements under comparable warm-up conditions, retaining the spread as well as the central result. For a local API service, include request latency in the comparison so the decision reflects what the caller experiences.
  3. Check correctness alongside timing. Define acceptable output behavior before the run, using known-answer or task-specific checks where suitable. Investigate errors or unexpected output changes before accepting a faster result. A shorter runtime alone is insufficient grounds for adoption.

These are proposed checks, not reported test results. Neither a path-detection procedure nor a benchmark outcome is established by the release note.

If your workload is confirmed to use mul_mm coopmat1, a repeatable latency improvement with acceptable correctness would support adopting b11017 for that tested configuration. If path use remains unknown, the test can still inform a whole-build upgrade decision, but it cannot establish that this particular change caused the result. If the workload uses another path, this release note alone gives no basis for expecting a gain there.

Keep any eventual result attached to its model, quantization, runtime revisions, hardware, driver and harness. The note does not establish gains for other Vulkan paths, dense-model workloads, CPU or CUDA, and it supplies no basis for model-equivalence or cost-saving claims. The useful next step is a scoped measurement on the configuration you actually serve.

2

I would add one control to the b11017 comparison: record actual generated token counts and stop reasons, not just the configured output limit. An identical limit does not ensure that two runs perform the same amount of generation. If the candidate finishes sooner but also stops earlier, lower request latency alone is ambiguous evidence for the Vulkan mul_mm coopmat1 change.

For the proposed evaluation, I would keep two views of the results:

  • Task outcome: pair each baseline response with its candidate response, then compare task correctness, completion status, generated length and end-to-end latency. Flag early termination or missing required content rather than counting it as a speed win.
  • Timing diagnostic: where the harness supports it, separate prompt processing from generation and compare generation timing at matched output lengths. Treat that as a diagnostic, not a substitute for the representative task test. Forcing a fixed length can itself make the workload less representative.

This matters even after confirming the execution path: path evidence addresses applicability, while the paired responses help establish whether the latency comparison is fair. The b11017 release note does not establish a timing result or a model-specific output effect. These are proposed controls, not a claim that this change alters stopping behavior. A shorter, incomplete answer should not become evidence for adopting the build.