sixi-asi 0.8: lossless speculative decoding for hybrid Gated-DeltaNet LLMs, measured against llama.cpp and ollama on the same tokens

Radek (Sixi) · 8 November 2026 · source, benchmark scripts and raw results

Hybrid models such as Qwen3.8 mix Gated DeltaNet layers, whose state is a fixed-size recurrence, with a few full-attention layers. That is great for long contexts and awkward for speculative decoding: a recurrent state cannot be truncated like a KV cache, so a rejected draft means either re-running the prefix or copying the whole state. sixi-asi is the server I built to make speculation on these models fast and provably lossless on a single NVIDIA unified-memory box. Today I am publishing it as a fork of my private research branch.

The decode loop

A step drafts k tokens, feeds anchor + k drafts to the target in one batch, accepts the longest matching prefix, and rolls the recurrent state back to the last accepted token in O(1) through llama.cpp's recurrent snapshot ring (n_rs_seq). No hundreds-of-megabytes state copy per token.

Three drafters feed that loop:

• an anchored suffix drafter (zero model cost; strong on code and repeated structure),
• the model's embedded multi-token-prediction head, chained for k=3,
• a DFlash2 block drafter: one pass over a block of mask tokens yields all drafts through a candidate lattice.

The draft length is not a guess. On this hardware every additional verify row costs about 17 ms, because the 48 recurrent layers scan the batch tokens sequentially; a measured sweep put the optimum at k=3 for both drafters, and k=7 clearly behind. The server picks k from that measurement rather than from the drafter's nominal block size.

Sampling is handled honestly. With temperature > 0 the server detaches the GPU argmax sampler, reads real logits for the k+1 rows, and runs rejection sampling against the exact top-k / top-p / temperature distribution the autoregressive sampler would use. A test with 4000 seeds puts the total-variation distance between speculative and autoregressive two-token continuations at 0.085, indistinguishable from the AR-vs-AR noise floor of 0.091; a deliberately wrong sampler scores 0.85.

Latency where agents feel it

Decode tok/s is the headline, but agent loops spend most of their wall time in prefill. Prefill speed is identical to llama.cpp on this box (about 266 tok/s on a 266-token prompt for both engines), so the only lever is not redoing it. sixi-asi keeps the live state when a request extends the previous conversation, rolls back within the snapshot ring when it diverges by a few tokens, and otherwise restores a composite snapshot (target plus drafter contexts) from RAM or NVMe. Second turn of a 1.1k-token conversation: 168 ms to first token instead of 3.4 s.

The chat side is equally unglamorous and equally necessary: a native Qwen3.8 template (tools, reasoning effort, thinking on/off, tool round trips) that renders byte-identically to llama.cpp's Jinja engine in 9/9 parity cases, reasoning budgets that force </think> and let the model answer instead of returning nothing, and schema-validated tool calls in the model's native <function=…> syntax.

Numbers

NVIDIA Thor developer kit (122.9 GiB unified LPDDR5X, ~273 GB/s, CUDA 13.2). Qwen3.8-27B Q4_K_M, 256 generated tokens, greedy, identical prompt tokens for every engine (ollama driven through its raw endpoint with the same rendered prompt), medians of 2–3 runs. llama.cpp commit a691b234 for every engine except ollama's bundled runtime.

Quality gate on the same prompts (reasoning effort low, greedy): sixi-asi 28/30 GSM8K, 16/20 HumanEval-lite, tool protocol and a 41k-character needle pass; llama-server AR 27/30, 15/20, pass, pass. The misses are the same problems for both engines.

What the numbers do not say

• Plain autoregressive decoding runs at about 60% of the bandwidth roofline in every engine measured, 99–100 ms per token in-process for sixi-asi and llama-server alike. The speed-up is accepted drafts per weight read, not different kernels.
• Every speculative engine, llama.cpp's own modes included, diverges from the autoregressive reference at the same near-tie tokens after a few hundred characters. The harness reports the divergence index instead of calling the output "bit-identical".
• The lead over ollama is 3.6%. A device-side state snapshot and acceptance-adaptive draft length are the next two items; both are measurable with the same scripts.

Every claim above ships with its script (scripts/ab_bench.py, scripts/intel_eval.py, scripts/template_parity.py) and the raw JSON under configs/benchmarks/2026-11-08/. If you can refute a number on your own machine, please open an issue.

Source, Apache-2.0: github.com/radeksixi/sixi-asi.