llama.cpp b10985: plan a paired RPC migration, not a cache-only patch
If a distributed llama.cpp deployment sees unexpected RPC cache growth, b10985 is worth evaluating—but not as an isolated server fix. Its release reports a two-node workload where activation transfers filled a worker cache to 1.4 TB after a day. Treat that as a workload-specific signal, not a capacity forecast.
The release adds a cache_flag byte to SET_TENSOR and bumps RPC_PROTO_MAJOR_VERSION because the wire format changes. Upgrade the communicating client/server pair together.
Runbook: inventory peers, builds, protocol expectations, cache ownership, and disk headroom. Stage both components in a canary. Stop if peers reject the protocol, transfers fail, or errors rise. Monitor cache bytes and growth per worker, disk headroom, hash/SET_TENSOR errors, disconnects, and inference failures. Expand only within declared budgets; otherwise halt and return to the last compatible pair.
This establishes a targeted fix and wire-format change, not a universal saving or production reliability result. Decide from your own topology and signals.
For a content pipeline that uses distributed llama.cpp to draft scripts or copy, I would separate the infrastructure canary from acceptance of the deliverable. The b10985 protocol change is a reason to coordinate the upgrade; it is not evidence that a production template still produces usable material.
My proposed handoff would attach a small, fixed content brief to the migration check: the same prompt template, model file, generation settings, and required output structure before and after the paired upgrade. Check for missing sections, truncated text, and extra material that would need manual cleanup. Record those separately from cache growth and RPC errors. A completed inference request should not be the sole acceptance criterion for a script that must fit a particular format.
For an in-progress batch, I would also record the client/server build pair alongside each output and mark the migration boundary in the batch manifest. That gives an editor a way to isolate material needing recheck without treating the entire batch as suspect. These are proposed workflow safeguards, not a claim that b10985 changes writing quality or that identical settings guarantee identical text.