Traction Runtime for SGLang

Upstream SGLang, pinned to a tested release, with a small published patch set: security backports, stability fixes, build hardening, and the live-tunable scheduler handlers we submit upstream. The OpenAI-compatible API, the native SGLang API, structured output, tool calling, multimodal inputs, speculative decoding, RadixAttention prefix caching, and tensor, pipeline, data and expert parallelism work as documented upstream. Model and quantization coverage follows upstream. Works inside llm-d SGLang pools and alongside NVIDIA Dynamo.

• LTS-1 cut from upstream v0.5.20; one new LTS branch per quarter
• Twelve months of backported security and stability fixes per branch; end-of-life published at cut and never shortened
• Hardened-profile, upgrade-path and performance-regression gates before any build is published
• Current track (Base): the latest two upstream releases, fix-forward
• Extended lifecycle and branch overlap (Premium)

• Pickle-based endpoints and code paths disabled
• API key required on every administrative route, including the internal-state route the tuner uses; allow-listed callers
• ZeroMQ inter-process sockets bound to localhost; multi-node transports on an authenticated mesh
• Non-root container image; append-only audit log of every configuration change, by actor
• Published hardening checklist so your security team verifies the profile rather than taking our word for it

• Upstream fixes backported, built and signed on every supported LTS branch; our own patch or mitigation carried when upstream has not responded, and submitted upstream
• Day-zero advisory stating whether the hardened profile already closes the class
• Public advisory tracker with status per LTS branch; coordinated disclosure with upstream
• Response targets per package in the product data sheet and subscription agreement

• Images signed with Sigstore cosign; SBOM and vulnerability-scan results attached to every build; reproducible from the public branch
• Authenticated registry for subscribers, public mirror for evaluation, offline bundles for air-gapped sites
• Admission-policy examples for Kubernetes and OpenShift that admit only what we signed

A controller that reads live scheduler signals and adjusts the settings that govern admission and batching: the running-request cap, chunked-prefill size and queue limit. Verify-or-revert with an SLO guard, canary, recommend-only, freeze and dry-run; sidecar plus Kubernetes operator with a TuningPolicy resource; no outbound network. Warm-started from the KPG Predictor's priors; sits under NVIDIA Dynamo Planner, llm-d and KEDA.

Aggregate metrics tell you that throughput fell. They cannot tell you why this request waited so long for its first token, what was in the batch when the drop happened, or why your system prompt keeps being evicted from the prefix cache. Traction Inference Reliability captures per-scheduler-step and per-request telemetry from inside the engine, stores it on your infrastructure, and attributes each request's latency to its cause: queue wait, KV pressure, prefill interference, chunking, GPU saturation. The tuner's recommendations, actions and verify-or-revert outcomes render in the same view, and derived metrics are bridged to Prometheus and OpenTelemetry so your existing dashboards keep working.

• Designed to a hard overhead budget of 1.5% throughput at steady state, enforced by the benchmark harness before any build ships
• No prompt or completion text is ever captured; prefixes are hashed with a per-deployment key
• Runs on-prem or air-gapped; no egress; license verified offline; behind your OIDC provider with viewer, operator and admin roles
• Redacted offline diagnostics bundle (tl-diag) so a support case can be worked without access to your cluster
• Engine telemetry hooks proposed upstream under Apache 2.0; the store, attribution and console are ours

Included in the Premium package as it ships.

• Business-hours coverage standard; extended hours in Enhanced and Premium; round-the-clock in Premium
• Published severity definitions; response targets per package in the product data sheet and subscription agreement
• Cases worked by the engineers who build the branches and contribute upstream

Traction Runtime for vLLM applies the same model to vLLM and is on the roadmap. A proprietary engine, Tread, is in design for a later premium variant. Neither is a product today.