llama.cpp MTP Fix – making Multi-Token Prediction reliable

If you run llama.cpp with Multi-Token Prediction (MTP) and the server prompt cache, you run into a bug that only shows up in real use: a branch from a stored checkpoint re-prefills silently from scratch – or the server ends with GGML_ABORT.

The cause is small but fatal: when a checkpoint is stored, llama.cpp saves the KV states of the target and draft model, but not the MTP carrier – the hidden row coupled to the KV boundary. This article describes the bug, the fix across five files, and the verification: from the HTTP test with Qwen3.8-27B, through the C++ unit test, to the mutant test.

Prerequisites

- a llama.cpp checkout at commit 2ca15f540476 (or a later commit where the patch applies cleanly)
- CMake, Ninja and a C++20-capable toolchain (verified with GCC 16.2.1)
- a target model with MTP layers plus a matching MTP draft model (here: Qwen3.8-27B, mtp-Qwen3.8-27B-Q4_0.gguf)
- a GPU backend for full verification (here: ROCm/Vulkan on an AMD Radeon AI PRO R9700)
- Python 3 for the HTTP verification test

Part 1: Applying the fix and starting the server

If MTP in server mode simply needs to work reliably with checkpoints, the practical flow is essentially this:

1. Get the patch

The patch ships as a single file:

- mtp-prompt-cache-fix.patch (reference version from julianmb/q38rocm, tag v1.5.0)

Apply the patch to the llama.cpp checkout in the project directory:

cd llama.cpp git apply --check ../mtp-prompt-cache-fix.patch git apply ../mtp-prompt-cache-fix.patch

--check before the actual apply is the deliberate step: the fix lives in tools/server and in common, exactly the areas upstream refactors often. If the check fails, the checkout is too new or too old for this patch.

The provenance of the patch at a glance:

- reference patch: julianmb/q38rocm, mtp-prompt-cache-fix.patch
- upstream adoption: ROCmFPX commit 0ef57fb8 („server: tolerate empty data_spec during checkpoint restore“, September 2, 2026)
- base commit of llama.cpp: ggml-org/llama.cpp, 2ca15f540476

2. Build llama.cpp

Standard CMake build with the test target:

cmake -S . -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DLLAMA_BUILD_TESTS=ON cmake --build build --target llama-server -j 8

For the full verification from the project directory:

./build.sh

The script builds llama-server and the unit test test-recurrent-state-rollback. The build string check then shows the version:

./build/bin/llama-server --version # => version: 0.5.0-dev (build 0, commit 2ca15f540476-mtp-fixed)

3. Start the server with MTP and checkpoints

The startup command for verification (Qwen3.8-27B, 262k context, RAM cache):

./build/bin/llama-server \ cold-long          32.03 s   26,458 prompt   0 cached          READY extension            0.47 s   26,478 prompt  26,459 cached     DONE branch               0.41 s   26,459 prompt  26,442 cached     BRANCH exact-repeat         0.21 s   26,459 prompt  26,455 cached     BRANCH RAM-cache-return     0.45 s   26,459 prompt  26,455 cached     BRANCH Status: PASS # without the fix (mutant: MTP serialization removed) cold-long          32.31 s   26,458 prompt   0 cached          READY extension            0.49 s   26,478 prompt  26,459 cached     DONE branch              31.96 s   26,459 prompt   0 cached         BRANCH Status: FAIL (branch: MTP branch/cache reuse missing: cached_tokens=0, required>=26330)

The extension variant (same prompt, slightly longer) uses the cache in both builds – it takes the regular KV path. The branch variant (a diverging prompt from the same checkpoint) is the decisive test: only it forces the server to actually restore the checkpoint. Without the fix: a full re-prefill in 31.96 seconds. With the fix: 0.41 seconds and 26,442 cached tokens.

The RAM-cache-return request additionally checks that an evicted and reloaded KV set takes the same path as the directly cached case. Again: 26,455 cached tokens.

5. C++ unit test and mutant test

Next to the HTTP test a C++ unit test (verification/test-mtp-checkpoint.cpp) exercises the server_slot methods directly, without the HTTP layer. It covers:

- MTP carrier serialization: get_state/set_state carry data_spec
- paired checkpoints: target, draft and MTP state are written together
- slot copy and slot reset: copy_prompt_to and reset handle the carrier correctly
- RAM restore and accounting: evicted states are accounted for correctly
- six cache failure routes and two checkpoint failure routes: broken entries throw instead of terminating the process

The result in the log file is a single line:

PASS: MTP carrier, paired checkpoints, slot copy/reset, RAM restore/accounting, six cache failure paths, and two checkpoint failure paths

The mutant test is the counter-proof: verification/mutants/libmtp-upstream-no-state.so builds out exactly the MTP serialization from the build, thereby reproducing the upstream behavior. The same HTTP test then runs against the same server – and it fails, as expected:

FAIL: MTP branch/cache reuse missing: cached_tokens=0, required>=26330

The unit-test equivalent: FAIL: MTP carrier must be serialized. A test that does not catch the mutant is no test – that is exactly what is checked here.

6. Result

With the patch applied and the server running on the patched version, the following holds:

- MTP in server mode works together with --ctx-checkpoints and --cache-ram
- a branch from a checkpoint loads the KV state instead of re-prefilling from scratch (0.41 s instead of 31.96 s in the test)
- an empty data_spec in a stored checkpoint is tolerated on restore and not aborted
- a broken checkpoint entry throws, is caught, and the slot is reset – the server keeps running
- the verification passes on three levels: HTTP test (PASS), C++ unit test (PASS), mutant test (detects the missing feature)

This is not a generic MTP performance feature, but a targeted fix for the case that would otherwise happen silently in the background: the checkpoint restore with MTP. Especially with long contexts and parallel requests sharing a prefix, exactly this path decides whether the cache helps or not.