Two Days, 57 Runs: Getting a 27B Local Model from 10 to 105 tok/s

I spent two days at the weekend asking my local AI the same question 57 times.

Not a lost weekend. I was timing the thing. Same model, same prompt, same PC humming under the desk, and I changed one server setting between batches to see what it did to the answer speed.

By Sunday night that 27B model had gone from about 10 tokens a second to 105. Nothing got re-quantised. Nothing got swapped out. The file on disk is the same one that produced Saturday night's slow numbers. All I touched was the serving configuration.

Here is where the speed was hiding, and which of my assumptions turned out to be wrong.

How the test works

tokbench.py is a small script. It asks an OpenAI-compatible server for 500 tokens of a fixed essay prompt at temperature 0, streams the reply, and writes down time to first token, total wall time, overall throughput and pure decode speed. Same prompt every run (mine happens to be an essay about air rifle target shooting, because that is what I had been thinking about), so the only variable is the server.

$ python3 tokbench.py --url http://192.168.1.248:8888/v1 \
    --tokens 500 --sheet 1AFI65Mbof2V3d9a42BP17N7OzUfGUW4l4ZXEAf8SYh8

def run_stream(url, payload, headers):
    req  = urllib.request.Request(url, data=json.dumps(payload).encode(), headers=headers)
    t0   = time.time()
    ttft = None
    n    = 0
    with urllib.request.urlopen(req, timeout=900, context=SSL_CTX) as r:
        for raw in r:
            line = raw.decode("utf-8", "replace").strip()
            if not line.startswith("data:"):
                continue
            data = line[5:].strip()
            if data == "[DONE]":
                break
            for ch in json.loads(data).get("choices", []):
                d = ch.get("delta", {})
                if len(d.get("content") or "") + len(d.get("reasoning_content") or ""):
                    n += 1
                    if ttft is None:
                        ttft = time.time() - t0
    return n, ttft, time.time() - t0

# overall     = n / total
# pure decode = (n - 1) / (total - ttft)

Strip the argument parsing and the optional Sheet logging and it is barely forty lines: open a stream, count the deltas, note when the first one lands, stop when the connection closes.

Fifty-seven runs across twelve sittings on 24 and 25 October, all appended to a Google Sheet called tokbench results. It has grown to 65 rows since.

I was not trying to score the model. I wanted to know how much speed was sitting there unused.

The headline

One model throughout: Qwen3.8-27B-Turbo-Brilliance-50X-RI-GSQ-RCO-IQ3_S, IQ3_S quant. Median overall throughput went from 10.5 to 51.5 tok/s. The best run went from 18.8 to 104.9. Pure decode went from 19.1 to 115.3. And the wait before the first word appeared fell from somewhere between 0.6 and 3.4 seconds to a steady 0.45.

Roughly five times on the median, five and a half at the top end, for changing settings and nothing else.

What is doing the work

The box under the desk is a Ryzen 9 5900X with twelve cores and twenty-four threads, an MSI RTX 4080 Super with 16 GB of VRAM, and 32 GB of DDR4. Not an AI server in any meaningful sense. It is a gaming PC that happens to be good at one job at a time.

Sixteen gigabytes of VRAM is the number that shapes everything else here. It decides the quant, the context length, and how much of the model has to fall back to the CPU. It is why every run in this post is a 27B at IQ3_S rather than something larger and more comfortable.

Consumer workstation prices have gone strange since the AI boom started a few years back. Faster cards exist. I am not buying one to win an argument with a benchmark script. This is the hardware I have, the limit is real, and the experiment is about squeezing what is there rather than replacing it.

The limit is not all bad, mind. Apart from the electricity, using this thing costs me nothing. The model, the server, the quantiser, the sampler, every piece of it is open source. I paid no licence fee for any of it, and I could not have run this experiment without it, so thanks to the people who write that software and hand it out for free.

The shape of it

The 57 runs clump into twelve sessions: back-to-back repeats a minute or two apart, hours of nothing in between. Read in order, those clumps tell the story better than any single number.

Day one is a plateau around 10 to 19 tok/s with one dead spot in the middle. Day two is a staircase: roughly 50, then 62, then a 105 ceiling that was still there three hours later when I came back to check.

The first run always lies

The biggest thing I got wrong early on was treating run one as a measurement.

Look at the six-run ladder from 17:49, ten seconds apart, identical prompt and settings:

40.9 → 51.1 → 61.4 → 85.1 → 104.9 → 104.4

Time to first token barely moves, 0.44 to 0.45 s the whole way, while total wall time falls from 12.2 seconds to 4.8. That is not six configuration changes. It is the same machine waking up: clocks ramping, caches filling. The 14:48 session walked the identical ladder a few hours earlier, which is how I knew it was not a fluke.

Two runs that do not count

The very first run on record was a 30-token probe at 1.3 tok/s, with a 19.5 second wait for the first token. That is a cold model load, not a benchmark.

The other is the 3.5 tok/s row on the 24th: 7.3 s to first token, 133 s total. That is what it looks like when something else on the box is fighting for the GPU mid-run.

Both rows are still in the sheet. Deleting your worst numbers is how you end up trusting a chart you shouldn't.

The setting that actually mattered

Of everything I fiddled with, one change did most of the work: the KV cache dtype, set to q8_0.

Measured outside this sheet, the box sat at 72 to 76 tok/s pure decode on the old dtype and reads 115 with q8_0. About 50 to 60% more speed for the cost of some VRAM. It has been the standing config ever since, and I am still mildly annoyed I did not try it first.

What about the MoE?

I had high hopes for Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF, mradermacher's quant. A mixture of experts with 17.8B active parameters ought to be the obvious throughput play. That is supposed to be the whole point of the architecture.

On day one it managed 8.0 to 12.9 tok/s, which put it in the same league as the plain Qwen3.8-27B-GGUF at 8.3. Not a triumph. In its defence, none of the cache tuning existed yet, so the comparison was not fair. I reloaded it with the same q8_0 KV cache and ran it again.

Tuning bought it about 1.7 times on the median: 8.2 to 13.8 tok/s, best 12.9 to 16.8. In percentage terms that is still a smaller lift than the dense model got — 4.9× on the median against 1.7× here.

It still finishes six times slower than the IQ3_S, and that gap is too big to wave away with quantisation or prompt length. As far as I am concerned the MoE hypothesis is dead on this hardware. Whatever the backend is doing with that GGUF, it is not getting the tuning the dense model gets. Re-quantising it, or poking at engine flags, is the next thing to try. I am not spending bench time on it until the multi-stream test is done.

Before you trust any of this

• tokbench counts streamed SSE chunks as tokens. Consistent, but an approximation, and it under-reports the fastest runs where several tokens land in one chunk.
• One prompt, 500 tokens, temperature 0, single stream. No concurrency, no batching, no long-context prefill. These are interactive chat numbers and nothing more.
• LAN HTTP overhead is inside the total. On loopback that gap would be smaller.
• A cluster of five or six runs is a warm-up ladder, not five independent samples. The top of a ladder is one data point.
• The server occasionally hangs on an identical repeat prompt and generates nothing for tens of minutes. Nudging the prompt slightly avoids it, so it smells like a prompt cache interaction rather than a hardware fault. Worth knowing before leaving a long agent run unattended on this endpoint.

Where it stands

10 tok/s to 105 on a 27B model, two days of poking at server settings, 0.45 s to first token. That is well past reading speed, and the bottleneck has moved from the serving stack to the model's own thinking.

One thing left open: concurrency. Single-stream throughput says nothing about what happens when two or three requests land at once, and that is the test that matters if I ever want to run agents against this box. Next weekend, probably.