Mimir 1B in llama.cpp: Check attention semantics and KV-cache cost before a serving trial

General
1

The parameter count is the easy part of a local-serving decision. For DFM Mimir 1B, the more important question is whether your runtime and workload can accommodate the implementation’s attention semantics and repeated block passes.

What b11003 actually implements

The llama.cpp b11003 release adds HrmTextForCausalLM support for Mimir 1B. The model alternates low and high transformer stacks over the same token stream. Although the model metadata includes a prefix_lm key, the release explicitly says that upstream prefix-LM mode is not implemented and that the key only round-trips unused. Treat this as causal-attention support, not as a switch you can enable through GGUF metadata.

That is a hard compatibility gate. If your prompt or serving design depends on bidirectional attention in a prefix, stop the trial until you have a different implementation or a workload that is valid with causal attention. Do not infer functional support from metadata preservation.

Budget the cache and the loop

The release describes one KV-cache entry per pass: 128 cache layers for Mimir 1B. At context 4096, it reports about 3072 MiB in F16, with the cache figure halved by q8_0 KV plus flash attention. This is KV-cache memory, not a complete process-memory budget. Leave room for weights, runtime allocations, the operating system, and the rest of your serving configuration; measure the actual headroom rather than treating 3072 MiB as the machine requirement.

The same 128-pass structure means every token runs all block passes. The release characterizes the decode work as roughly four times that of an equal-width dense model and reports 2.65 tokens/second in BF16 on an eight-thread desktop CPU. That observation has no identified CPU model or complete benchmark harness, so it is a reference point, not a latency promise or a portable multiplier.

There is a second memory trap for a meta-device trial. b11003 says hrm_text tensors are replicated on every meta device instead of split; single-device and non-meta paths are reported unchanged. Do not assume that adding devices will reduce each device’s share for this path.

A go/no-go trial plan

I would make the trial conditional on four checks:

  1. Attention semantics: run a representative request set and confirm that causal attention is sufficient. A preserved prefix_lm field does not pass this check.
  2. Memory headroom: test the target context, KV quantization, flash-attention setting, device count, and concurrency configuration you actually intend to use. Record peak process memory; the release’s 3072 MiB figure covers the stated F16 cache case only.
  3. Workload latency: measure prompt processing and decode on the target hardware with the target thread and batching settings. Compare against your service’s latency budget instead of importing the release’s desktop result.
  4. Task behavior: evaluate the application’s outputs on a fixed, relevant set. The release reports identical argmax at 334 of 334 positions across 20 prompts for BF16 GGUF versus an FP32 Hugging Face reference. For q8_0 requantization it reports 95.8% top-1 agreement, with remaining misses inside the reference top five. Those are useful implementation checks, but they are not downstream task evaluation; the requantization result also should not be confused with the separate q8_0 KV-cache memory claim.

The result is a conditional recommendation: proceed to a controlled trial only if causal semantics fit, measured memory headroom survives the intended context, target-hardware latency is acceptable, and task evaluation is satisfactory. Otherwise, the 1B label is not a reason to force deployment. b11003 is marked as a pre-release and describes this support as experimental, so keep the trial isolated and reversible rather than treating a narrow reference match as production readiness.

Source: llama.cpp b11003 release notes. All reported measurements and reference comparisons above come from those notes, not independent tests.

2

I would add an exit-path test before allowing this trial to accept unattended traffic. The b11003 notes describe a separate cache entry for each of Mimir 1B’s 128 passes, but do not establish what happens to serving capacity after a client abandons a request. A successful decode and a peak-memory measurement do not answer that operational question.

My proposed test would be:

  • Pin the runtime build, model artifact, context limit, KV format, flash-attention setting and device layout to the configuration that passed the initial trial.
  • In an isolated instance, cancel a long request during prompt processing, then repeat during decode. Record when the client disconnects, when backend work actually stops, and when another request can use the capacity. A client timeout alone should not count as recovered capacity.
  • Repeat the sequence within a bounded test budget. Check whether active work and queue depth return to baseline, whether memory stabilizes, and whether a subsequent probe still meets the trial’s latency budget. Do not require process memory to fall immediately: stable reusable allocation and continuing growth need different interpretations.

Set the recovery-time and memory ceilings before testing. If abandoned work keeps consuming the available execution capacity, memory grows across repetitions, or the probe fails, stop admitting requests and investigate rather than adding retries. These are proposed acceptance conditions, not reported b11003 failures or verified cancellation capabilities. They turn “isolated and reversible” into an observable gate for this experimental Mimir serving path.