If you run llama.cpp with Multi-Token Prediction (MTP) and the server prompt cache, you run into a bug that only shows up in real use: a branch from a stored checkpoint re-prefills silently from scratch – or the server ends with GGML_ABORT.

The cause is small but fatal: when a checkpoint is stored, llama.cpp saves the KV states of the target and draft model, but not the MTP carrier – the hidden row coupled to the KV boundary. This article describes the bug, the fix across five files, and the verification: from the HTTP test with Qwen3.8-27B, through the C++ unit test, to the mutant test.

If you run several local agents (Hermes, Pi, cron jobs) against a single llama-server, you hit the same problem: every session gets its own slot, every slot its own prompt cache – and shared prefixes still get re-prefilled from scratch over and over.

The session-slot router is a test setup that turns this around: a small Python proxy on port 8081 knows the session ID of every request, assigns each session to a slot and keeps it there via LRU. This article describes the architecture, the Pi and Hermes integration, the slot logic, and the limits of this first iteration.

Two things are killing the speed: a bug in llama.cpp that loses the MTP carrier in the cache checkpoint – and it happens even without any parallelism. And when pi and Hermes run at the same time, llama.cpp sometimes compares the prompt cache wrong: it does not realise the prompt is already cached and reprocesses it from scratch. The two companion articles cover exactly that:

MTP-Fix for llama.cpp →

Session-Slot-Router →