sixi-asi 0.11: 32 concurrent streams on one Jetson Thor at 170 tok/s, in 27 GB instead of 70
sixi-asi 0.10 served four requests at once on an NVIDIA Jetson AGX Thor and stopped scaling at sixteen: 129 tokens per second in total, whether sixteen or thirty-two clients were connected. Version 0.11 reaches 154 with sixteen and 170 with thirty-two, uses 27 GB of context memory for those thirty-two streams instead of 70, and no longer makes an agent wait minutes behind other agents' long prompts. The model is the same Qwen3.8-27B; the default weights are a better file.
| 0.10 | 0.11 | |
|---|---|---|
| 32 requests at once, aggregate | 129 tok/s | 169-171 tok/s |
| 16 requests at once | 129 tok/s | 151-154 tok/s |
| 4 requests at once | 94.2 tok/s | 103.5 tok/s |
| 16 agents with a 17.8k-token prompt each: wait for the next turn | up to 442 s | under 3 s |
| Context memory for 32 streams | 70.2 GB | 27.1 GB |
| Default weights, KL divergence to BF16 | 0.025 (Q4_K_M) | 0.008 (UD-Q4_K_XL v3.1) |
| One stream, Q4_K_M | 45.7 tok/s | 45.5 tok/s |
Why sixteen streams were the ceiling
On Thor a forward pass of Qwen3.8-27B is bound by memory bandwidth: every pass reads 15.8 GB of weights, about 85 ms, almost regardless of how many rows are in it. Concurrency is cheap only while all streams share one pass. They did not. A profile of the 32-stream server showed three things:
- Split passes. llama.cpp keeps sequences in one micro-batch only when their ids are consecutive. One stream that arrived a few milliseconds late left a hole, and the pass became two or three, each with its own full weight read. The server now waits for stragglers in proportion to the last pass time (20%, 4 to 40 ms), and a missing stream no longer splits the pass.
- State copies. 48 of the model's 64 layers are Gated DeltaNet layers with a recurrent state of 3.1 MB each. Every pass copied the states of all sequences into a contiguous buffer and back. The kernel now reads them in place.
- Snapshots. To roll back a rejected draft token, the library kept a snapshot of the recurrent state after every verified row: 144 MiB per row, per sequence.
A log instead of snapshots
The third item is the larger change. The recurrent update of a Gated DeltaNet layer is a function of the previous state and four small per-row inputs: key, value, gate and beta. Instead of storing the state after each row, 0.11 stores one state per sequence and the inputs of the last few rows. A rollback of n rows replays the kept rows on top of the base state, inside the same kernel that processes the new rows.
Context memory for 36 sequences (32 streams and 4 snapshot slots) drops from 70.2 to 27.1 GB. Rolling back a whole batch, or twice in a row, was impossible with the snapshot ring and is now legal. Before this change a 32-stream configuration with a deeper ring asked for 92 GB and took the machine down; the server now sizes a context before allocating it and refuses a configuration that does not fit, saying what would.
Agents with long prompts
The scenario: sixteen agents, each with its own 17.8k-token document, each asking two follow-up questions while other agents are still loading theirs. With 0.10 scheduling a follow-up queued behind every load in progress: the second turn waited 223 s (median) and the slowest follow-up 442 s. In 0.11 long prefills run outside the request lock and hand the device to waiting requests between chunks: 2.8 s median, 2.9 s at worst. The cold loads pay about 8% of the total time for the interleaving.
Quality decides the default
Unsloth's Dynamic v3.1 files changed the trade. Against the BF16 logits on the same corpus:
| Weights | KL divergence | One stream | 32 streams |
|---|---|---|---|
| Q4_K_M | 0.0248 | 45.5 tok/s | 168.9 |
| UD-Q4_K_XL v3.1 | 0.0077 | 44.0 tok/s | 170.5 |
Three times closer to the full-precision model for 3% of the single-stream speed, so it is the default. The rule for this project from now on: a better file wins when it costs less than about 10% of the speed. Both files pass the same gate (GSM8K 28/30, HumanEval 19/20, tool calls, needle) and are level on 40 AIME problems (70.0% per sample; 82.5% and 80.0% with a majority of three), which says more about the size of that test than about the files.
What did not work
- Longer drafts with many streams. The pass gets wider, the row cost returns, the aggregate falls.
- Two other drafters. 41.2 and 41.5 tok/s against DSpine's 46.
- Plain 4-bit integer weights in the layout tensor cores read best (one scale per 128 columns): KL divergence 0.081 without calibration, ten times the default file; 5 bits reach 0.019. The layout needs calibrated rounding and wider sensitive tensors before a kernel for it is worth writing.
- Voting on the server (
"vote": truewithnsamples, stops once the majority is decided) is in the release and correct, but on eight hard problems it saved 7% of the time, not the third one might hope for: the questions where it matters are the ones where the samples disagree until late.
Every 0.11 number above was measured on the published build, with the library built from the clean pinned llama.cpp commit and the eight patches in the repository. The scripts (scripts/concurrency_bench.py, scripts/agents_bench.py, scripts/intel_eval.py, scripts/hard_eval.py) and the raw JSON (configs/benchmarks/2026-11-12/) are in the repository. The 0.10 figures beyond four streams and the 0.10 agent waits were measured on the 0.10 code before this work started.
Source, Apache-2.0: github.com/radeksixi/sixi-asi. Previous write-up: sixi-asi 0.9.