The engine is fast. The settings around it are what waste the GPUs.
Early accessSGLang's scheduler settings are fixed at launch; the workload is not. The Traction Tuner adjusts them live against what the engine is actually seeing, with the guard rails an enterprise needs before it lets software change production settings. An open-source harness measures the result, so the only numbers we publish are ones you can reproduce.
Early access with design partners.
Why static configurations leave capacity on the table
The settings that matter are set once.
How many requests may run at once, how large a prefill chunk is, how deep the queue may go. They are chosen at launch for a traffic shape that will not be the traffic shape at three in the afternoon.
Nobody re-tunes in production.
Re-tuning means a restart, a change ticket and a risk. So teams pick a safe setting, protect the latency target with headroom, and accept that headroom they never use is idle GPU.
The signals exist; nothing acts on them.
Queue depth, arrival rate, length distributions, prefix-hit ratio, KV occupancy and retractions are all observable inside the engine. The tuner is the thing that reads them and does something.
How it works
What it adjusts.
Live settings first: the running-request cap, chunked-prefill size and queue limit, through the upstream internal-state API and the validate-apply-readback handlers we submit upstream. Restart-required settings (KV memory fraction, parallelism, attention backend) follow through rolling reconfiguration with drain, one replica at a time.
How it decides.
A rule-based controller with hysteresis, cooldown, step limits and a change budget, so it moves deliberately and does not oscillate. It is warm-started from the KPG Predictor's priors for the model and hardware: the predictor sizes before deployment, the tuner refines after. Workload fingerprinting into regimes and per-regime optimization follow in the advanced tuner.
How it stays safe.
- Verify-or-revert on every change, with an SLO guard: if p95 time-to-first-token or inter-token latency degrades inside the verify window, the change is rolled back and logged
- Canary on one replica before fleet-wide
- Recommend-only mode with human approval, which is how most teams start
- Freeze switch and dry-run
- Every action in an append-only audit log: actor, setting, old and new value, reason, outcome
How it fits your stack.
- Sidecar plus Kubernetes operator with a TuningPolicy custom resource; Helm; offline signed license; no outbound network
- Operates inside a replica, beneath fleet planners: it consumes NVIDIA Dynamo metrics and exposes a capacity signal to Dynamo Planner, llm-d and KEDA rather than competing with them
- Engine-agnostic settings registry; a vLLM backend is on the roadmap
Sizing before deployment
The KPG Predictor, our patent-pending sizing model, forecasts throughput, time-to-first-token and cost per million tokens for a model on a given GPU and traffic shape from the model's kernel performance graph, without running the workload. On this site it appears in two places: it sizes your fleet during the Runtime Assessment, and its forecast is the tuner's starting point.
The benchmark harness
Open source at launch. The only numbers on this site will be ones it produced.
| Trace | Upstream default | Expert-tuned static configuration | Predictor-sized static configuration | Predictor-sized with the tuner running |
|---|---|---|---|---|
| Chat | steady · bursty | steady · bursty | steady · bursty | steady · bursty |
| Retrieval-augmented | steady · bursty | steady · bursty | steady · bursty | steady · bursty |
| Long-context | steady · bursty | steady · bursty | steady · bursty | steady · bursty |
| Code | steady · bursty | steady · bursty | steady · bursty | steady · bursty |
| Reports | Goodput at SLO (tokens per second delivered inside the p95 TTFT and TPOT bounds), cost per million tokens and SLO-violation minutes | |||
The harness replays traces for chat, retrieval-augmented, long-context and code workloads at steady and bursty load, three runs per cell, and compares four configurations: the upstream default, an expert-tuned static configuration, a predictor-sized static configuration, and predictor-sized with the tuner running. It reports goodput at SLO (tokens per second delivered inside the p95 TTFT and TPOT bounds), cost per million tokens and SLO-violation minutes. It runs from a clean checkout against your own cluster, carries the overhead gate for Inference Reliability telemetry and the regression gate for every LTS build, and every benchmark post ships with model, hardware, driver, CUDA, container digests and the exact command.
Until the first post publishes, we describe the tuner qualitatively: it keeps GPUs busy. We will not put a percentage on this page that the harness has not produced.
Harness source[GitHub URL at launch]Validated hardware
| Accelerator | Build lane | Harness coverage |
|---|---|---|
| NVIDIA H100 | CUDA 12.x and 13 | All four trace types; tuner and telemetry overhead gates |
| NVIDIA L40S | CUDA 12.x and 13 | All four trace types; tuner and telemetry overhead gates |
| NVIDIA A100 | CUDA 12.x | Chat and retrieval traces |
| NVIDIA B200 | CUDA 12.x and 13 | Large mixture-of-experts profiles |
| AMD ROCm | On request under a paid engagement | On request |
Model coverage follows upstream SGLang. An unlisted model that runs on upstream runs on Traction Runtime.
Bring your model and your traffic
A Runtime Assessment runs the harness on your workload, sizes it, applies the hardened profile, and leaves you a measured baseline you can defend. The fee is credited against a subscription.
Results
No published results yet.
Results are published one workload at a time.
Performance figures on this site, when present, are produced by the open-source Traction Layer benchmark harness and published with model, hardware, driver, CUDA and container digests and the command that reproduces them. We publish no performance figure the harness has not produced.