Assumed Healthy vs Observed Healthy

For thirteen days in late summer 2026, one model on Cloudflare’s Workers AI platform had a strange property. Send it a very long prompt and it answered quickly, fluently, and with a success status. The answer just had nothing to do with your prompt. It was generated from the leftover state of other customers’ requests: their prompts and partial outputs.

Cloudflare disclosed this on October 26. It counted 155 affected responses between August 27 and September 9, every one to a prompt longer than 681,280 tokens, all on a single model, GLM-5.3. A researcher at Knostic reported it late on September 8. Cloudflare took the model offline about 15 hours later, 40 minutes after declaring an incident, and restored it, fixed, the same day.

A good deal has been written about this already, and most of it makes the same sound point: a 200 status hid the problem. I was not involved and have spoken to nobody who was. What this piece adds comes from reading the engine’s public source code, and I mark where I am inferring. Three things stand out that the public accounts I have found leave unexplained: why one half of the system said yes while the other said no, a safety check that existed and was defeated, and an internal reply that contradicted what the client was told and was thrown away.

They all point at one distinction. A system is assumed healthy when the checks you have are passing and you conclude it is fine. It is observed healthy when something has looked at the property that matters and seen it hold. This incident lived in the gap between the two. Health was assumed, not observed. Cloudflare’s own post uses the word twice: once for what the team assumed about the engine, and once for what one machine assumed about another.

What happened

Cloudflare served GLM-5.3 in a split layout. Prefill GPUs read the prompt and build the model’s key-value (KV) cache. That cache is shipped over RDMA to decode GPUs, which generate the response. A router sends each request to one worker of each kind. The cache lives in a shared pool of fixed-size pages, and for performance the pages are not wiped between requests.

1. Two limits disagreed. The context length was declared in configuration: 1,048,576 tokens. The KV pool size is not a setting you declare. By default the engine derives it at startup from the GPU memory left after loading weights. On the prefill nodes it came out at 681,280. Cloudflare knew. By its own account it expected an error for prompts the engine could not process, saw its long-context tests return success, and assumed the engine was coping with the difference.

2. Each worker enforced its own limit. In the upstream source, a worker checks a prompt twice: at the front door against the context length, and in its scheduler against a cap derived from its own pool. The prefill cap was about 681,000 tokens. The decode workers had more room and accepted the same prompt. This is why the two halves could disagree at all.

3. The prefill worker did reject the request. Its rejection handler recorded the error as pending and swapped the prompt for a single placeholder token, so the dead request could drain cheaply through the normal machinery and report its error on the way out. On a single node, that works.

4. Nothing on the split path read the pending rejection. Cloudflare describes the admission queues as checking whether a request was finished, not whether it was rejected. In the source, the request is queued, a one-token forward pass runs, and one page of cache is sent with a completion signal. The step that turns a pending rejection into an error is never reached.

5. The receiver trusted “complete.” Decode had reserved pages for the full prompt. It received one page and a success signal that, in the source, carries no length, page count or checksum. It began generating, and attention read across everything it had reserved. One token slot held the placeholder. The rest held what earlier requests had left there.

Simplified from upstream SGLang before v0.5.20, and not the literal source:

# Scheduler, on each worker, using that worker's own pool-derived cap
if len(req.origin_input_ids) >= max_req_input_len:
    req.set_finish_with_abort(error)    # to_finish = ABORT(400); origin_input_ids = [0]
    add_request_to_queue(req)           # still queued, so the error can drain out

# Prefill worker's capacity guard, reached after the swap
if len(req.origin_input_ids) > pool_tokens:    # 1 > 681,280 is False
    reject(req)
sender.init(num_pages=1)

# Prefill result handler in split mode
send_kv_chunk(req, last_chunk=True)     # the pending abort is never promoted here

# Decode worker, which accepted the same prompt under its own larger cap
reserve(pages_for(full_prompt_length))
wait_until(transfer_status == SUCCESS)  # nothing to compare the amount against
generate()                              # attends over everything it reserved

Look at the second block. The prefill worker had a guard comparing prompt length to pool size, and its failure branch was the correct, fail-closed one. By the time it ran, the prompt was one token long. A defensive check was defeated by a defensive transformation upstream of it. I verified this in upstream v0.5.18 and v0.5.19. Cloudflare runs its own patched build and has not said which version it was on.

What each signal showed

Cloudflare has not said what its production monitoring showed. What can be reconstructed is how an affected request looked from outside, and what the launch tests checked.

None of these would have flagged the problem, and several pointed the wrong way.

Scale matters too, and this is my own arithmetic. 155 responses in thirteen days is about twelve a day. No error rate or latency percentile moves on twelve requests a day. A violation this rare is invisible to aggregates even in principle.

The person who noticed was reading the output. Knostic’s researcher saw file paths from a machine that was not his, then put a unique marker at both ends of a large prompt and asked for it back. At 662,162 tokens the marker returned after 68.9 seconds. At 700,168 it did not, and the response took 7.1 seconds. The broken request was about ten times faster than the healthy one.

Assumed, not observed

Status codes, latency, throughput and token counts describe the motion of a pipeline. The property that matters most in multi-tenant inference is provenance: a response must be a function of this caller’s input and the model’s weights, and nothing else. Nothing in the table measures that. Availability can be measured from outside by anyone. Correctness can only be measured by something that knows what the right answer was.

Reliability engineers will recognize this as gray failure, and it is not new. What this incident shows unusually clearly is the step where it happens. A passing result was taken as evidence about a property the test never examined. And the warning was available: an error you expected and did not get is as much an anomaly as an error you got and did not expect.

Four patterns that generalize

Failure paths that borrow the success path. Letting a dead request flow through as a cheap stub is an optimization. It is safe only while every stage the stub passes knows it is dead. Add a stage or a topology, and the drain path becomes a way in.

A pending state that a new consumer did not read. “Rejected, not yet reported” was a deliberate intermediate state, and the single-node path knew how to finish it. The split path was a newer consumer and did not. A two-phase state is a contract with every path that can carry the object.

Declared limits versus emergent limits. The context length was a setting. The pool size was a consequence of quantization, parallelism and memory fraction. A limit derived at runtime has to be asserted against the limit promised to customers, from the running system, on every change. Cloudflare’s own verdict: “This is the check we did not enforce.”

The two halves disagreed, and one answer was thrown away. This is my inference from the upstream source. There, the prefill worker’s own reply for such a request reports a prompt of one token, and the router returns only decode’s reply. The rejection handler also writes an error-level log line. If Cloudflare’s build behaved the same way, each affected request left an internal record contradicting what the client was told. Cloudflare does not say whether such records existed or were examined. The rule holds regardless: when two components hold different views of one request, the disagreement is worth alerting on.

The report that read as a quality problem

On July 6, seven weeks before launch, another SGLang user filed an issue describing the same path, with a patch. A pull request followed two days later and reviewers were assigned. Review in mid-September found defects, and it is still open. The issue got no human reply and was closed by an inactivity bot on September 5. A separate change that closes the hole reached the main branch on September 6 and first shipped on September 18. No release contained a fix during the exposure window.

The report described garbage output from uninitialized cache, and on a single-tenant cluster that is what it is. On a shared cluster the garbage is someone else’s prompt. As Rajesh Beri argued shortly after the disclosure, on a multi-tenant cluster a report like that is a security report. Put more generally: in a system with shared state, output unrelated to the input is an isolation problem until proven otherwise.

This path was also not the only one. SGLang’s own post describes a separate race, in which late RDMA writes from an aborted request land in pages already handed to someone else. It says that race leaves no error or log line.

What to do about it

• Name the boundary. Write down the properties whose violation is worse than downtime. If nobody can state them, nobody is observing them.
• Treat a pass you cannot explain as a failure you have not found yet. If the design says a request should fail and it succeeds, find out why before shipping.
• Put an echo marker in the tests you already run. A unique token at both ends of the prompt, asserted in the output, would have turned the existing long-context test red. At this size it costs about a minute, which suits a deployment gate and not continuous probing.
• Assert derived limits against promised limits, from the running system. Do not infer them from a passing test.
• Give the receiver an independent expectation. It knows how much it reserved and should not start on less. Cloudflare’s draft patch did this. Upstream chose to reject earlier and offer an optional checksum.
• Make violations harmless, and count the cost. Zeroing memory on release turns a leak into garbage, though the request still succeeds with a wrong answer. Ownership tags checked at read time go further. Cloudflare built a prototype, measured under one percent overhead on a mid-sized model, and switched it off in mid-August for reliability problems, about two weeks before this launch.

That last fact is the uncomfortable one. A correctness check competes with availability for the same budget, and it loses unless the boundary it protects has been named.

What is not public

• What production monitoring showed, and which engine build was running.
• How the 155 were counted, how many customers’ data appeared in them, and whether those customers were told.
• Whether anyone exploited it. Cloudflare found no evidence of deliberate exploitation, which is an absence of evidence.

Where this leaves us

I cannot judge the launch decision from outside. Once the report was verified, the response was fast, and the disclosure is unusually specific about what was assumed and why that was wrong. For anyone running SGLang in split mode, the fix shipped in v0.5.20 with a cleaner admission-time rejection in v0.5.21. Cloudflare recommends v0.5.22 or later, and v0.5.23 is the latest at the time of writing.

The question to take home is short. What is the one property your users cannot tolerate you breaking? Which signal turns red when it breaks? And where are you assuming health because nothing has told you otherwise?

Sources

• Cloudflare, “Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang,” October 26, 2026.
• SGLang / LMSYS, “SGLang security update: improving request isolation,” October 26, 2026.
• Knostic, “Hallucination or Cross-Tenant: When an LLM Answers With Another User’s Context,” October 27, 2026.
• Cloudflare, “Smaller, faster, safer: running Kimi and GLM at scale,” August 3, 2026.
• SGLang issue #30233 and pull requests #30551, #36988, #38704, #38935, #39500, #46435.
• SGLang source at release tags v0.5.18 and v0.5.19, under python/sglang/srt/.
• Rajesh Beri, “Cloudflare’s GLM-5.3 Leak Exposed Other Users’ Data in 155 Responses,” October 29, 2026.
• Huang et al., “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems,” HotOS 2017.