You’ve turned on --enable-mixed-chunk, so replies that are already streaming keep getting tokens while a long prompt is processed in chunks. (Without it, they don’t.) To protect them further you halve chunked_prefill_size and restart the server. An hour later you open the dashboard. The tail of inter-token latency looks worse than yesterday’s. Was it the change?
Maybe. But a before-and-after comparison on a live server has at least three ways of misleading you, and none of them show up as an error. This post walks through each one, using what stock SGLang v0.5.20 exposes, and then the one setup that sidesteps most of them.
1. The restart reset more than one thing
In stock SGLang v0.5.20, chunked_prefill_size and prefill_max_requests can’t be changed on a running server, so trying a new value means a restart. A restart doesn’t just apply your setting. With default settings, it also empties the prefix cache. (A persistent tier, such as HiCache with a storage backend, can survive a restart, but that is opt-in.)
That matters more than it sounds. With the cache warm, a chat turn that shares a long system prompt or conversation history with earlier requests only prefills the new tokens. With it cold, the first request for each shared prefix pays for the whole thing, and long conversation histories have to be rebuilt one by one. So the first stretch after a restart has more prefill work per request than the day before, at the same traffic. Any change you made to prefill behaviour is being judged on a heavier workload than the one you made it for.
You can see when this has passed. With --enable-metrics, the ratio of rate(sglang:cached_tokens_total) to rate(sglang:prompt_tokens_total) gives a steady cache hit ratio. Don’t judge until it’s back where it was before the restart, and then compare windows where it’s about the same. The sglang:cache_hit_rate gauge and the #cached-token field of the Prefill batch log line are per batch, so they jump around. Average them if you use them.
2. The traffic moved, and your metrics can’t tell you
Your “before” is yesterday afternoon and your “after” is this morning. Those are different workloads: different numbers of people, and different prompts. So the first thing to check is whether the load is comparable. Here most of the obvious metrics don’t help, because the change you made moves them too:
- Queue depth (
sglang:num_queue_reqs) depends on how fast the server drains the queue, and draining speed is exactly what chunk size controls. A smaller chunk drains long prompts more slowly, so the queue grows even at the same traffic. - Throughput (
sglang:gen_throughput) moves with the change for the same reason. - SGLang’s own request and prompt-token counters (
sglang:num_requests_total,sglang:prompt_tokens_total) are incremented when a request finishes, not when it arrives. They count completions, so they’re downstream of the scheduler as well.
What you need is the arrival rate: how many requests arrived, and how many prompt tokens they brought. Independent users send requests at their own pace, whatever your chunk size is. Count arrivals in front of the server, where they enter: your load balancer, gateway or router already logs every request. If all you have is SGLang’s metrics, you can get a rough request count: add completions, aborted requests (sglang:num_aborted_requests_total) and the growth in the waiting and running counts over the window. Those counts are sampled. They miss requests still in the tokenizer and requests the server rejects, for example when its queue is full, which is exactly when load is high. Treat the result as approximate. There’s no equivalent for prompt tokens.
Then compare like with like. If arrivals rose a lot between your two windows, a worse tail tells you nothing about the change. One caveat: clients that wait for each reply before sending the next aren’t independent. Their arrivals follow completions, so a slower server also lowers their arrival rate.
3. “Better” depends on why you changed it
There’s no single score for a server like this, because the settings trade one group of users against another. With mixed chunk on, a smaller chunk helps the people already mid-reply and costs the people waiting for their first token. A larger chunk does the reverse. If you judge either change on a single number, you’ll call half of your good changes bad.
So decide before you change anything what the change is for, what should get better, and what you’re prepared to lose:
| You changed it to… | It worked if… | Watch that it doesn’t… |
|---|---|---|
| Protect running replies (smaller chunk, with mixed chunk on) | The tail of sglang:inter_token_latency_seconds drops at comparable load | Push the time to first token past what your users accept |
| Drain a backlog of prompts (larger chunk, wider prefill batches) | sglang:queue_time_seconds and the tail of sglang:time_to_first_token_seconds{is_streaming="true"} drop at comparable load | Push the tail of inter-token latency past your target |
Write those down before you look at the dashboard. It’s easy to find a number that improved after the fact. (Read the inter-token histogram with care if you serve non-streaming requests, run speculative decoding, or stream more than one token at a time. Non-streaming requests share the same histogram and report in batches of tokens. When several tokens arrive together, it records their average gap, so a stall spread across a batch of tokens looks smaller than it felt.)
The better test: don’t compare across time at all
All three problems come from comparing two different stretches of time. If you already run several replicas behind a load balancer, you can avoid that. Change the setting on one replica, leave the others alone, and compare them over the same hours. They see the same traffic mix and the same time of day, and after the first stretch the changed replica’s cache is as warm as the others’.
Two things to get right:
- Routing. A cache-aware router sends requests with shared prefixes to the same replica, and it’s the default policy in SGLang’s model gateway. That’s good for serving and bad for a fair comparison, because the two replicas then see systematically different requests. Use a random or round-robin split for the test.
- Run it long enough to cover the traffic you care about. A morning of chat says little about the afternoon’s long documents.
This is the closest thing to a controlled experiment a live server gives you. It is what we’d recommend to anyone tuning by hand.
Where a tuner fits
Doing this by hand works for the occasional change. It doesn’t work when the right setting depends on a traffic mix that changes during the day. That’s the case we built Traction Tuner for, and it runs into the same three problems on a single server:
- Traction Runtime, our hardened SGLang distribution, accepts live changes to these two settings within bounds set at launch: the chunk size can’t be raised above its launch value. So there’s no restart and no cold cache. The new value applies from the next batch, but the measurements right after a change still mix in the old setting, so the tuner lets a window pass before judging anything.
- Runtime adds arrival counters, so the tuner can compare offered load directly. When a change looks harmful and load moved in the direction that would explain it, the tuner leaves the change in place rather than revert it on bad evidence, unless the change broke the latency target.
- Every change records what it was for, and the tuner checks it against that, or against what it has already measured at that load.
It starts in observe-only mode and backs off if someone sets a value by hand. Like you, it assumes mixed chunk is on: shrinking the chunk only helps running replies when it is. We haven’t published measurements of it yet. When we do, they’ll come with the exact commands, hardware and versions.
If you want the background, why chunk size only trades one group of users against another once mixed chunk is on, and why a full KV cache is a separate problem, it’s in our ebook, Why long prompts stall everyone else.
Everything here describes SGLang v0.5.20 and the current Traction Tuner. Traction Layer AI is not affiliated with, sponsored by or endorsed by the SGLang project.
I’m an AI engineer at Traction Layer AI. I build Traction Runtime, our hardened SGLang distribution, and Traction Tuner. If your logs tell a different story, I’d like to hear it: find me on LinkedIn or write to hello@tractionlayer.ai.