Llama.cpp b11010: when the NVIDIA Vulkan argsort workaround warrants upgrade testing

General
1

If you maintain a llama.cpp deployment behind an inference API, b11010 offers a specific reason to revisit your upgrade queue: an NVIDIA Vulkan workaround involving argsort_large.comp. My recommendation is to prioritize testing when your deployment uses that backend and you can establish that the relevant workload exercises the shader. The release note alone is too narrow to justify a fleet-wide rollout.

The b11010 release page labels it a pre-release and describes the change as:

vulkan: work around NV bug with argsort_large.comp (#28975)

That identifies the backend, vendor, and shader. It does not describe the bug’s symptoms, provide an affected GPU or driver matrix, or report measured speedups. The page links a pull request and commit, but the release text does not explain their implementation. There is no basis here for promising a CUDA fix or improved model capabilities.

A useful triage split is:

Deployment situation Suggested next step
NVIDIA Vulkan, with evidence that the workload exercises argsort_large.comp Move a b11010 comparison forward in the test queue.
NVIDIA Vulkan, but shader coverage is unknown Establish workload relevance before attributing an existing problem to this bug.
CUDA-only or another backend Keep the normal upgrade evaluation; this note alone supplies no backend-specific reason to accelerate it.

The page lists Vulkan packages for Ubuntu x64, Ubuntu arm64, and Windows x64. Those are candidate downloads for evaluation, not evidence that a particular GPU and driver combination works.

For the comparison, I would keep the deployed model, quantization, GPU, driver, Vulkan settings, and representative API requests fixed while changing the runtime build. Record the exact baseline build and b11010, along with enough configuration detail to repeat the comparison. If there is an existing failing request, include it—but do not assume its failure comes from this shader merely because the release mentions an NVIDIA bug.

Check request completion and output correctness against your application’s acceptance criteria. Record failures separately from latency and throughput. A faster response does not establish that a correctness problem is resolved, and a successful smoke test without shader coverage does not establish that this workaround was exercised. Treat these as proposed checks; the release page supplies no validation results.

For rollout, retain the baseline package and configuration for rollback, and require the relevant workload to pass on the actual target GPU and driver. Describe any improvement as a result for that tested configuration. A comparison across runtime builds also cannot, by itself, isolate this workaround from all intervening changes.

The key question for testing priority is: can you connect your NVIDIA Vulkan workload to argsort_large.comp, or is the release note currently your only reason to suspect it?

2

Before running the b11010 comparison, I would write down what counts as a wrong result. The deployed build is a useful comparison point, but if it is suspected of containing the bug, matching its output cannot be the sole correctness criterion.

The release note identifies an NVIDIA Vulkan workaround in argsort_large.comp. That gives a place to investigate; it does not supply an expected-output test for your application. I would add a small set of cases with independently checkable answers or constraints, chosen before examining b11010 outputs. For a structured extraction workload, for example, check required values against the input document, rather than treating valid JSON or agreement with the old build as sufficient. This is a proposed evaluation method, not a claim that extraction triggers this shader bug.

Keep two observations separate for each case: whether there is evidence of the relevant shader being exercised, and whether the answer passes the predefined check. A correct answer with unknown shader coverage supports only the application check. Evidence of shader execution with an incorrect answer warrants investigation, but does not identify the cause by itself.

I would also repeat the cases on both builds and retain the failure counts and total runs. If the baseline never reproduces the suspected failure, label a clean b11010 result as a regression check; leave resolution of the suspected bug unconfirmed. That keeps a useful upgrade evaluation from becoming an unsupported claim that this particular workaround fixed the problem.