For an application considering a switch from BF16 Mimir 1B GGUF weights to q8_0 in llama.cpp, the b11003 conversion checks are a reason to evaluate the quantized candidate, not enough evidence to accept it. The important distinction is between matching a reference model’s highest-ranked token and preserving the outcomes your application needs.
The llama.cpp b11003 release notes add support for HrmTextForCausalLM (DFM Mimir 1B). They report two fidelity checks:
| Comparison | Reported result | What remains unproven |
|---|---|---|
| BF16 GGUF versus FP32 HF reference | Identical argmax at 334/334 positions across 20 prompts | Agreement outside that sample, identical logits, and application quality |
| q8_0 weight requantization versus HF reference | 95.8% top-1 agreement; all remaining misses inside the reference top five | Whether the disagreements matter for the target task |
These are source-reported results, not independently replicated measurements. The notes do not provide prompt contents or position-selection details, and they do not separately specify the q8_0 sample denominator. I would not attach the BF16 sample count to the q8_0 percentage as though that denominator were explicit.
Top Five Is Not an Acceptance Criterion
An argmax match says that the highest-ranked token agrees at a checked position. It does not say that the entire probability distribution is identical. Likewise, a disagreement that remains inside the reference top five is still a disagreement with its first choice, not proof of an equally acceptable answer.
For application acceptance, 95.8% token agreement is neither a 95.8% task success rate nor a measured 4.2% quality loss. The release notes provide no mapping from these token choices to completed task outcomes. The BF16 result has the same boundary: perfect agreement on the reported sample does not establish that BF16 itself meets your application’s requirements.
The notes attribute the q8_0 discrepancies to accumulated error over 128 sequential blocks. That is the source’s explanation, not a causal result independently established here. It helps frame what to investigate; it does not quantify the consequences for a particular workload.
A Paired Test Before Substitution
My recommendation is to turn the weight change into a controlled application evaluation:
- Define success before comparing outputs. Select representative requests, including important failure cases. Choose task-level measures appropriate to the application, such as exact field correctness for extraction or rubric-scored answers for a constrained question-answering workflow. Set an absolute quality floor and an allowed regression relative to BF16; do not borrow a threshold from the token-agreement percentage.
- Change the weights, not the whole serving configuration. Compare corresponding BF16 and q8_0 artifacts using the same pinned llama.cpp build, input formatting, context settings, hardware/backend, KV-cache configuration and decoding settings. Record artifact identities. Keep flash-attention settings fixed too, so a cache or backend change does not confound the weight comparison.
- Score completed outputs and inspect disagreements. Evaluate both variants on the same requests. Check whether BF16 meets the absolute quality floor before using it as the acceptance baseline. Inspect new q8_0 failures in high-consequence cases, not just the aggregate score. For stochastic generation, use repeated runs under the same sampling policy rather than treating one pair of outputs as conclusive.
- Make acceptance conditional on that workload. Accept the substitution only if the predefined task criteria are met with enough evidence for the application’s risk. Otherwise retain the baseline if it is adequate, or investigate further. Measure any desired latency or memory benefit separately; fidelity statistics cannot establish it.
This is an evaluation proposal, not a claim that those tests have been run or that Mimir supports every example workflow adequately.
Keep the Implementation Boundary Visible
The b11003 notes call this implementation experimental and support causal attention only. Upstream prefix-LM mode is explicitly unimplemented, so an acceptance test on this path cannot validate a workflow that requires that mode.
Also keep two uses of q8_0 separate. The notes’ cache-memory statement concerns q8_0 KV cache plus flash attention, not q8_0 model weights. Their desktop-CPU decode figure is labeled BF16. Neither supplies a q8_0 weight speedup or quality measurement.
For this Mimir substitution, the defensible conclusion is narrow: b11003 reports strong sampled token agreement, with some top-1 changes after weight requantization. Whether those changes are acceptable is an application-level question that the release’s checks leave open.