Session-Slot-Router – distributing local AI sessions cleanly across slots

If you run several local agents (Hermes, Pi, cron jobs) against a single llama-server, you hit the same problem: every session gets its own slot, every slot its own prompt cache – and shared prefixes still get re-prefilled from scratch over and over.

The session-slot router is a test setup that turns this around: a small Python proxy on port 8081 knows the session ID of every request, assigns each session to a slot and keeps it there via LRU. This article describes the architecture, the Pi and Hermes integration, the slot logic, and the limits of this first iteration.

Prerequisites

- a running llama-server with multiple slots (here: http://127.0.0.1:8080/v1)
- Python 3 for the router (here: 3.14)
- the router listens on 0.0.0.0:8081, locally http://127.0.0.1:8081/v1
- agents that can attach a session ID and a client type (Pi extension, Hermes middleware)
- sqlite3 for slot assignment and the event log (no prompts)

Part 1: Setup and startup

The router is a single Python module (router.py) that start-router.sh launches in the background with nohup, printing its PID and writing to router.log. The SQLite database lives at router.sqlite regardless of the working directory. Only one instance can hold port 8081 – a second start reports the conflict in the log.

1. Starting the router

The start script does three things: launch the process, print the PID, name the log file. To follow along:

./start-router.sh
tail -f router.log

The log contains date/time with timezone, HTTP method and route (without query), agent_name, agent_session_id, client_type, the client-supplied id_slot, the detected slots, the actual slot, the action (including any evicted previous session), the response status and rejections. Prompts, tools, request bodies and API keys are deliberately not logged.

2. How slot assignment works

The router forwards method, route, query, headers and body without agent_session_id unchanged to llama-server on port 8080. If a JSON body contains an agent_session_id:

- the router reads /slots and selects the assigned slot
- new sessions take free slots, otherwise the least-recently-used one
- agent_name, agent_session_id and client_type are removed, id_slot is overridden
- in-flight requests block their slot; if all are busy, the new request waits

The client_type is pure protocol information and does not influence slot selection – for old clients it stays NULL. A cli or print type does not prove that a human rather than a script triggered the request.

3. Integrating Pi

The Pi extension (~/.pi/agent/extensions/pi-session.ts, copy under plugin-pi/) attaches the session ID and client_type only when the base URL explicitly uses port 8081. ctx.mode is passed through unchanged (tui, print, rpc or json). The test invocation:

pi --provider llama-slot-test --model Qwen3.8-27B-S1

The model is configured in ~/.pi/agent/models.json; the extension loads automatically.

4. Integrating Hermes

plugin-hermes/ is a general Hermes plugin with an llm_request middleware – deliberately not a model provider. It attaches the session ID only when the base URL uses port 8081. For cron platforms or a matching cron session ID it sends cronjob, otherwise the platform name or unknown.

A copy lives under ~/.hermes/profiles/<PROFIL>/plugins/session-slot-router/ and is listed in <PROFIL>'s plugins.enabled; it is not installed globally under ~/.hermes/plugins/. Important: the plugin only augments main-agent requests; auxiliary model requests do not pass through this middleware hook.

Session IDs matching cron_<12-hex Job-ID>_<YYYYMMDD_HHMMSS> are rewritten to cron:<Job-ID> so that runs of the same job send the same routing ID. All other session IDs are left unchanged.

5. Limits of this first iteration

- Eviction happens without asking and without guaranteed safety – an older context may be re-read. Direct requests to port 8080 bypass the router entirely.
- The router does not read existing contexts from the slots; after a restart of router or llama-server a stored assignment proves nothing about preserved KV state.
- No authentication: on 0.0.0.0:8081 the router is reachable from all network-accessible interfaces as long as the firewall allows it. For network access, restrict the UFW rule to trusted source IPs.
- No guaranteed resumption of aborted streaming requests and no cap on wait time when all slots are busy.
- Forwarded methods are GET, HEAD, POST, PUT, PATCH, DELETE and OPTIONS; WebSockets and bodies without Content-Length are not implemented.

6. Tests

The tests live under tests/ and are run from the project directory:

python3 -m unittest discover -s tests -v

They cover, among others, passthrough without a session ID, routing/LRU with session ID, streaming, HTTP methods, the SQLite migration, terminal logging, and the Pi and Hermes plugin metadata. Pi and <PROFIL> were tested against the live router and llama-server.