Does tool calling work through the server?
Six short requests through the OpenAI-compatible endpoint with two tools
declared: does the server return structured tool_calls, use a tool result,
make two calls in one turn, stay quiet when no tool is needed, stream a call, and copy a
path argument exactly. A fail is usually the parser, not the model. Produced by
agentic.py tools.
Agentic loop
A coding-agent transcript replayed turn by turn: after every tool result
the server is asked for the next step. Warm lets the server reuse what it
already computed for the previous turn; cold defeats that so every turn is
prefilled from scratch. The gap between them is what prefix caching is worth. Produced
by agentic.py loop.
Colours identify configurations. Solid lines are warm, dashed are cold. Synthetic is a 600-word system prompt and sixteen "read this file" turns of ~1.2K tokens each, 128 tokens out; typical and heavy replay real sessions from the usage profile below, with the context and output sizes they actually had. Turns whose context does not fit the server's window are skipped and counted.
Time to first token per turn
What the agent waits before the next step starts arriving, as the context grows turn by turn. Lower and flatter is better.
Context per turn
Tokens the server was given at each turn. The synthetic transcript grows in even steps; real sessions jump at compactions and resumes.
Every run
| Configuration | Shape | Agents | Mode | Turns | Over window | TTFT turn 1 | TTFT last turn | TTFT p95 | Context at end | Cache hit | Turns / min |
|---|
What real sessions look like
Token counts are the provider's; the local tokenizer lands within
roughly a tenth of them. Produced by usage.py from the
claude_code.api_request events; no prompt text or identity leaves the logs.
Context per request
What the server has to hold for one request: the conversation so far, cached or not. Bars are 20K-token bins.
Output per turn
Tokens generated per request, in powers of two. The synthetic loop forces 128.
Tools called
| Tool | Calls | Failed | p50 | p90 | max |
|---|