A flat small-shape benchmark is not enough to dismiss a Metal FWHT kernel improvement. The useful measurement question is whether the suite includes a case where kernel execution—not launch overhead—dominates.
The llama.cpp b11059 notes, on a page marked Pre-release, give a concrete example. Their branchless butterfly-select change in the Metal FWHT simdgroup kernel showed a reported gain on one larger Hadamard matmul, but no visible difference at the shapes already in the performance suite. For engineers evaluating a local inference backend for an API service, that is a reason to check benchmark coverage before interpreting a flat result—not a promise of faster requests.
What the comparison actually measured
These are upstream-reported results on M5 Pro, not independently reproduced measurements:
| Case | Reported timing | What the source says |
|---|---|---|
| Existing performance-suite shapes | 1.6–3.9 µs per operation, against a 1.6 µs launch floor | The change is not visible there. This range is not a before/after pair. |
| Hadamard matmul, block 512, 65,536 rows | 1324.6 µs before; 1285.0 µs after | Kernel execution rather than launch dominates; the notes describe a 3.0% gain. |
The larger-case comparison used five interleaved A/B rounds with the first discarded. The notes describe the changed kernel as faster in every round. They also report trying FOR_UNROLL on the same loops: its delta changed sign between rounds, no benefit was reported, and it was left out.
The measured change here replaces a ternary in the butterfly shuffle stages. Although the release notes also discuss F16 input support, the timing gain is reported under the branchless change, not as a measurement of F16 versus F32 input.
Keep two measurement questions separate
My recommendation is to retain small cases and add cases where kernel execution dominates, reporting them separately:
- Small cases: does the changed implementation alter operation time near the measured launch floor? A flat result belongs in the report, but cannot by itself rule out a gain at a larger shape.
- Kernel-dominated cases: is the change measurable when more of the operation time is spent doing kernel work? Block 512 with 65,536 rows is the source’s example to investigate, not a universal cutoff for Apple hardware.
For a follow-up A/B comparison, hold the hardware, input type, shape, and timing method fixed, and record the exact baseline and changed builds. Interleave runs, declare any warm-up exclusion before comparing results, and retain the per-round timings rather than publishing only a winning aggregate. Those are proposed measurement practices; the notes do not document all those controls. Their five-round design is a starting point, not proof that five rounds will resolve every small effect.
If the larger case improves while the small cases stay flat, report a shape-specific kernel gain. If the measured delta changes sign across rounds, keep the conclusion inconclusive pending more measurement rather than selecting the favorable round. The FOR_UNROLL result illustrates that distinction without proving unrolling can never help.
Where the conclusion stops
The supplied timing passage has no raw per-round results, uncertainty estimates, full benchmark command, named model workload, or end-to-end serving measurement. “Faster in every round” is useful reported evidence, but is not enough here to establish statistical significance.
The supported decision is to broaden the FWHT benchmark’s shape coverage—not to assign a 3% latency improvement to an API endpoint, extrapolate to another chip, or decide whether to deploy this release. A serving claim needs a separate named-model workload measurement. Until then, keep the kernel result and the request-latency question on different lines of the report.