vLLM on one RTX 5090

Decode throughput

Tokens per second, single stream. Higher is better.

KV cache pool

Tokens the card can hold at that configuration's context length, as reported by vLLM at startup. Rows keep the order of the chart above so the two can be read together; the scale is different, which is why this is a second chart and not a second axis.

All measurements

The same runs in full. Every charted value is here too.

Concurrency sweeps

How a configuration behaves as simultaneous requests pile up. Every request is forced to emit a fixed number of tokens (256 unless the sweep says otherwise), so throughput reflects the server rather than how much the model felt like saying. Prompt and output length vary by sweep — filter on them below. Produced by bench.py.

Colours identify configurations, not models. Narrow the sweeps with the filters below.

Total throughput

Tokens per second across all in-flight requests. Higher is better.

Time to first token, p95

What a caller waits before anything comes back. Lower is better.

Every level

Did the flag change the model?

Two configurations of the same model are fed the same fixed text and asked, at every token, what they thought was coming next. The further apart those predictions, the more the flag changed the model rather than its speed. Produced by quality.py; what these numbers mean.

Read every row against the noise floor for its model — the same server disagreeing with itself. At or below the floor means indistinguishable from running the reference twice.

Mean KL divergence, teacher-forced

Lower is closer to the reference. Hatched bars are noise floors.

Every comparison

Noise floor per configuration

The same scoring request sent repeatedly to one server. Nothing changed between requests; this is how much the server moves on its own.

What did not work

Recorded because these are the configurations you would reach for first.