Speed up local AI: Qwen & llama.cpp, the reliable way

Two things are killing the speed: a bug in llama.cpp that loses the MTP carrier in the cache checkpoint – and it happens even without any parallelism. And when pi and Hermes run at the same time, llama.cpp sometimes compares the prompt cache wrong: it does not realise the prompt is already cached and reprocesses it from scratch. The two companion articles cover exactly that:

MTP-Fix for llama.cpp →

Session-Slot-Router →

The problem in brief

Qwen 3.8 (27B) runs fine on the R9700 out of the box – but two things are holding the system back:

• The MTP bug: When saving a checkpoint, llama.cpp drops the MTP carrier of the prompt cache – after which Multi-Token Prediction silently fails. The error is independent of load and shows up even with a single, solo-running request.
• The parallelism problem: As soon as two agent harnesses (pi and Hermes) hit the same llama-server at the same time, llama.cpp sometimes does not compare the new requests against the prompt cache correctly – it fails to see that the prompt is already in the cache and reprocesses it in full. This is exactly where the Session-Slot-Router steps in: it assigns each request a unique session slot before the cache comparison, so the cache hit is guaranteed and no second prompt-processing pass takes place.

Both problems need their own solution – and both are in the two companion articles: the MTP-Fix for llama.cpp and the Session-Slot-Router.

My setup

• GPU: AMD Radeon AI PRO R9700 – because Nvidia cards are unaffordable
• Harnesses: pi and Hermes
• Main LLM: Qwen 3.8 (27B, Q4_K_S, 150k context)
• Fast LLM: Tiel Coder 1.5 – a MoE model for fast tasks
• Server: llama-server as a local OpenAI-compatible API

What the fixes bring

With stable MTP and sessions that are assigned up front, the system delivers what the hardware can give: the cache holds – even across checkpoint restores – and the two harnesses run in parallel without one of them redoing the other’s work. This article ties the whole thing together; the fixes themselves live in the two companion articles.

Test results

All models tested on the same hardware (R9700, llama-server, Hermes). “Main” and “Fast” mark my current choice.

llama-server start script (Thinking & MTP)

#!/usr/bin/env bash
set -euo pipefail


model_name=Qwen3.8-27B-UD-Q4_K_S
draft_file=mtp-Qwen3.8-27B-Q4_0.gguf
proj_file=mmproj-F16.gguf


# #########################  parallel #########################
if [[ $# -ne 1 || ! "$1" =~ ^[1-9][0-9]*$ ]]; then
  echo "Verwendung: $0 <parallel>" >&2
  exit 2
fi

parallel="$1"

if [[ "$parallel" -eq 1 ]]; then
  ctx_single=262144
elif [[ "$parallel" -eq 2 ]]; then
  ctx_single=150000
else
  echo "Erlaubt sind nur 1 oder 2 als <parallel>." >&2
  exit 2
fi

# #########################  path #########################
model_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"

model_file="$model_name.gguf"
model_path="$model_dir/$model_file"

if [[ -n "$proj_file" ]]; then
  proj_path="$model_dir/$proj_file"
else
  proj_path=""
fi

if [[ -n "$draft_file" ]]; then
  draft_path="$model_dir/$draft_file"
else
  draft_path=""
fi

# #########################  start #########################
nohup llama-server \
  --model "$model_path" \
  --alias "$model_name" \
  --mmproj "$proj_path" \
  --spec-draft-model "$draft_path" \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-ngl 999 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0 \
  --host 0.0.0.0 \
  --port 8080 \
  --parallel "$parallel" \
  --kv-unified-per-slot "$ctx_single" \
  --kv-unified \
  --no-cache-idle-slots \
  --n-predict 32768 \
  --gpu-layers 999 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn auto \
  --temperature 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --jinja \
  --reasoning-preserve \
  --reasoning-budget -1 \
  --verbosity 4 \
  > "$model_dir/llama-server.log" 2>&1 &

echo "llama-server gestartet: PID $!"


#   --chat-template-file "$model_dir/chat_template(sharp).jinja" \