Qwen 3.8-27B
the architecture, from scratch.

System overview

Qwen 3.8-27B is a decoder-only transformer with four departures from the textbook design: a hybrid attention backbone, multimodal rotary position encoding, trained-in multi-token prediction, and a native vision front-end. The remainder of this document derives each departure and the engineering decisions it implies.

The three pressures that shape the architecture

Every constant in this model is a response to three simultaneous pressures, and holding them in mind makes the rest of the document legible. The first is long context: the model must attend over 262k tokens natively and stretch to 1M, which makes the attention key-value cache the dominant cost (a 262k-token softmax layer holds an $O(T)$ cache that, multiplied across 64 layers, would alone consume tens of gigabytes of VRAM per active request). The second is multimodality: image and video tokens must enter the same residual stream as text without a separate decoder or a retrained position embedding, so that one set of layers handles every modality. The third is serving economics: at 27B parameters the model is sized to fit a single 80 GB device, so every constant (head counts, FFN width, KV precision, the ratio of linear to full-attention layers) is tuned against a fixed memory ceiling rather than against quality alone.

The central tension is that softmax attention is the only component with perfect recall but the only component whose memory grows with sequence length. Linear attention and state-space models collapse the per-token cost to a constant by writing each token into a fixed-size recurrent state, but that compression is lossy: a needle buried in a 200k-token haystack is not guaranteed to survive the write. Qwen 3.8 resolves the tension not by choosing a side but by interleaving the two regimes: compressing the bulk of the sequence with linear layers and periodically re-reading the raw context with a full-attention layer. The architecture is, in this sense, a schedule for when to remember exactly and when to remember approximately.

Reading the dataflow, stage by stage

The diagram reads left to right as a convergence into a single bus and then a vertical stack. Two input paths enter on the left: a SigLIP-style vision encoder that turns images and video frames into patch embeddings (~0.9B parameters, §07), and a text tokenizer that maps raw bytes to integer IDs in a 248,320-symbol vocabulary (§02). The vision path is narrower in the diagram but heavier per token, since a single image can cost thousands of tokens, so before merging it passes through a projector and 2×2 spatial merge that folds each 2×2 block of patches into one token, a 4× reduction that trades spatial resolution for sequence length. Text needs no such reduction; it is already discrete.

Both paths converge into the unified stream, a single tensor $x \in \mathbb{R}^{T \times 5120}$ carrying text and vision tokens on equal footing. From here on, no layer distinguishes a text token from an image token: the modality boundary lives only in the position encoding (MRoPE, §07) and the embedding path, never in the backbone. The stream enters the 64-layer hybrid stack, the amber column, arranged as 16 repetitions of $[3 \times \text{DeltaNet} + 1 \times \text{Attention}]$. The stack's output returns to an LM head that projects the final hidden state back to a 248,320-way distribution over the vocabulary, plus auxiliary MTP heads that predict several tokens ahead for self-speculative decoding (§08).

The residual stream as the central bus

The single most important structural fact about this model is that every layer is additive into one residual stream. A layer does not overwrite its input; it computes a correction and adds it, which is what makes the architecture composable: each of the 64 layers is a small residual function whose output is summed back into the bus, so the hidden state at layer $l$ is the input plus the accumulated contributions of every preceding layer. The norm-before-residual convention (Pre-RMSNorm) means each layer normalizes a copy of the stream for its own computation but writes its un-normalized update back, so the stream's magnitude grows steadily rather than exploding.

Two consequences follow and recur throughout the document. First, the mixer (DeltaNet or attention) and the FFN each read a normalized view of the same stream but contribute independent updates, so they specialize: the mixer routes information across positions while the FFN transforms the representation at a single position. Second, because the stream is never reset, the model's effective depth is the full 64-layer sum. A token's final representation is its embedding plus a layered set of corrections, and removing or reordering layers changes the sum. That is why the 3:1 schedule (§03) is not cosmetic: the order in which approximate and exact memory layers are stacked changes what the stream converges to.

The four departures from the textbook transformer, each of which gets its own section:

• 1. A hybrid attention backbone: 64 layers arranged as 16 identical super-blocks of [3 × (Gated DeltaNet + FFN)] + [1 × (Gated Attention + FFN)]. 75% of layers use linear attention with a constant-size recurrent state, so their per-token cost does not grow with context; 25% use full softmax attention, whose KV cache grows with sequence length but which guarantees exact retrieval. The two are not interchangeable: the linear layers compress the stream cheaply and the full-attention layers periodically re-anchor it against the raw context, preventing the associative drift that a pure linear stack accumulates. This is the single most important design decision in the model (§03–05).
• 2. Interleaved multimodal RoPE (MRoPE): rotary position encoding normally assigns each token a single 1D position, which breaks for images that have a 2D layout and video that has a 2D+time layout. MRoPE splits the rotary phases into three sections (mrope_section: [11, 11, 10]) that carry the temporal, height, and width axes independently, so a 2D image grid or a video timeline slots into the same 1D token sequence as text without retraining tricks or separate position tables (§07).
• 3. Trained-in Multi-Token Prediction: the model ships with extra heads that predict several tokens ahead in one forward pass. At inference these heads act as a draft model that is the same model: the backbone proposes multiple continuations, verifies them in a single pass, and accepts the longest prefix that checks out, amortizing the per-token cost of long autoregressive generation. Because the heads were trained jointly with the backbone rather than distilled afterward, their predictions share the backbone's distribution and reject rarely (§08).
• 4. A native vision front-end: a SigLIP-style vision tower turns images and video frames into patch embeddings, which a projector then maps into the LM's 5120-wide residual stream. The backbone therefore never sees "vision" as a special case: image tokens are tokens, processed by the same 64 layers as text, which is why the model can interleave reasoning over figures and prose without a separate vision decoder. This is a first-class input path, not a bolt-on adapter (§07).

Parameter budget

Before reading a single equation, account for where the 27 billion parameters actually live. The audit below is computed live from the config; every number in §03–06 derives from this breakdown.

The budget breaks into three tiers of very different size. The per-layer FFN is the largest single sink: a 17,408-wide SwiGLU has three projections over the 5120-wide hidden state (§06), costing roughly 269M parameters per layer and, across 64 layers, the majority of the model. FFN width is the primary lever on per-token capacity. The mixer (DeltaNet or Gated Attention projections, §04–05) is surprisingly small by comparison, because attention parameters scale with head dimension and head count rather than with sequence length, and the linear layers reuse a fixed recurrent state instead of growing a projection. The vocabulary is the third tier and the most top-heavy single object: the embedding matrix alone is 2.54 GB in BF16, and with an untied LM head the input and output boundaries each carry that cost, so they outweigh any single backbone layer. Toggle the vision tower (~0.9B) and the MTP module (~1.3B) below to see how much the auxiliary paths add on top of the dense core.

From raw Unicode bytes to 248,320-dimensional discrete spaces

Everything downstream in the 64-layer backbone consumes a discrete sequence of integer token IDs $\mathbf{x} \in [0, 248320)^{T}$. The byte-level BPE tokenizer defines the model's vocabulary contract, compressed sequence length, and structured protocol format across 119 languages.

Byte-level BPE, from the bottom up

Byte-pair encoding builds the vocabulary bottom-up rather than assigning one token per word. The algorithm starts from the smallest possible alphabet (raw UTF-8 bytes, 256 symbols) and scans the training corpus for the most frequent adjacent pair of symbols, merges that pair into a single new token, adds it to the vocabulary, and repeats. Each iteration shrinks the corpus by one merge and grows the vocabulary by one token; after roughly 248,064 merges the vocabulary reaches its working size. Because the base alphabet is bytes, every possible Unicode string is representable: there is no <unk> token and no out-of-vocabulary fallback, which is what lets one tokenizer serve 119 languages, source code, and mathematical notation simultaneously.

The merge sequence is fixed at training time and baked into the tokenizer file, so the same string always segments identically at inference. Two consequences matter for the rest of the architecture. First, the tokenizer, not the model, decides the sequence length $T$ for a given input, so its compression rate directly sets the attention FLOPs and KV cache the backbone must pay. Second, the vocabulary size fixes the dimension of the embedding lookup and the final LM-head projection, two of the largest single tensors in the model.

Why 248,320 and not 248,064: padding for the Tensor Core

The merge schedule produces 248,064 learned tokens, but the vocabulary is declared as 248,320, a padded size chosen as $3{,}880 \times 64$. The 256 extra rows are never used as tokens; they exist so the vocabulary dimension is an exact multiple of the Tensor Core matrix-multiply-accumulate (MMA) tile. The embedding lookup and the LM-head projection are GEMMs with the vocabulary as their contracting dimension, and on modern NVIDIA Tensor Cores a matmul only hits peak throughput when that dimension is a clean multiple of the tile (64 for BF16); a partial tail forces either masked lanes or a separate epilogue, both of which waste bandwidth on a tensor this large. Padding costs 256 unused rows (~2.6 MB of BF16) and buys peak embedding and head throughput on every forward pass, so the trade is overwhelmingly in favor of padding.

The embedding matrix $W_{\text{emb}} \in \mathbb{R}^{248320 \times 5120}$ is the model's only bridge from discrete integers to continuous geometry: each token ID selects one row, and that 5120-vector is the initial residual-stream value before any layer runs. The LM head is its mirror, a projection from the 5120-wide residual stream back to a 248,320-way distribution, and in Qwen 3.8 it is untied: a separate learned matrix rather than the transposed embedding. Input and output boundaries are therefore symmetric and together hold roughly 5 GB of BF16, more than any single backbone layer. The widget below shows the segmentation that feeds this lookup.

Fertility: why vocabulary size is a serving lever

Fertility is the number of tokens a tokenizer emits per word, and it is the single number that ties the tokenizer to the backbone's cost. Sequence length $T$ enters attention as $O(T^2)$ compute and the KV cache as $O(T)$ memory, so a script that tokenizes at 2.5 tokens/word rather than 1.3 multiplies attention FLOPs by roughly $2.5^2 / 1.3^2 \approx 3.7$ and the KV cache by $2.5 / 1.3 \approx 1.9$. A small vocabulary forces high fertility on non-Latin scripts because their graphemes cannot share merges with Latin text; a large vocabulary dedicates merge budget to those scripts and compresses them toward Latin rates.

The 32k → 151k → 248k vocabulary progression across Qwen generations is an economic argument. Each step costs a larger embedding and LM-head tensor but buys lower fertility on Chinese, Hindi, Arabic, Thai, and Khmer, which in turn cuts the quadratic attention cost and the KV cache footprint at long context, exactly the regime Qwen 3.8 is built for. The matrix below measures that compression directly across the three vocabulary sizes.

Special token protocol catalog: boundary & agentic contracts

The upper address space of the vocabulary, IDs near 248,000, is reserved for control tokens, and these are exempt from BPE merging. That exemption is the point: a delimiter like <|im_start|> or a boundary like <|vision_start|> must be a single atomic symbol that the tokenizer never splits or merges with its neighbors, or the protocol it carries would be corrupted by segmentation. Reserving them as fixed, single-token IDs guarantees that atomicity for free.

Read together, the control tokens are the protocol layer the model speaks: they mark chat roles for post-training, bracket test-time reasoning traces that serving backends parse and strip, anchor visual patch tokens in the MRoPE coordinate system, and wrap structured tool-call JSON for agentic execution. Each entry below maps a token to the downstream subsystem that consumes it:

The output of this section is a fixed contract: a raw input of any modality becomes an integer sequence $\mathbf{x} \in [0, 248320)^T$, which the embedding lookup turns into the initial residual stream. From here the backbone takes over, and the first question it faces is how to mix 48 constant-cost linear layers with 16 exact-retrieval attention layers without the stream drifting. That is the subject of §03.

64 layers, two complementary memory regimes

The fundamental structural innovation of Qwen 3.8 is the $16 \times [3\times \Delta + 1\times A]$ periodic super-block: three Gated DeltaNet linear-attention layers for constant-memory bulk compression, followed by one Gated Attention layer for lossless exact recall, each accompanied by a 17,408-width SwiGLU FFN.

Two memory regimes, one stream

The "hybrid" in the backbone is a pairing of two ways a layer can remember the past, each with an opposite cost curve. A linear-attention (DeltaNet) layer writes each incoming token into a fixed-size recurrent state (a matrix $S$ of constant shape) and reads from that same state. Its memory is $O(1)$ per layer regardless of sequence length, and its per-token compute is constant; the price is that the state is a compression of history, so old or low-weight information is gradually overwritten and cannot be recalled verbatim. A full softmax attention layer instead stores every token's key and value in a cache of size $O(T)$, and a query reads any past token exactly by softmax-weighted lookup. Its memory grows with context and its compute with $T^2$, but it retrieves perfectly.

No single regime is correct at 262k tokens. Pure linear attention is cheap but loses needles in the haystack; pure attention recalls everything but its cache alone would dominate an 80 GB device. The backbone's answer is to run the cheap, compressive regime on most layers and slot the exact regime in periodically, so the stream is usually being integrated cheaply and is occasionally re-anchored against the raw context. The next equation fixes that schedule.

To see why "compression" is the right word, look at what the recurrent state actually holds. Full attention stores $T$ separate key–value pairs and retrieves one by inner product; a DeltaNet layer instead accumulates their summed outer product into a single fixed matrix $S \approx \sum_t v_t k_t^{\top}$, and a query reads back $q^{\top} S$. The $T$ rank-one pairs have been collapsed into one $d \times d$ matrix of constant size, and that is the compression. It is why memory is $O(1)$, and why recall is lossy: a sum of outer products cannot be inverted back into the individual pairs, so a query that should match one specific past token instead retrieves a weighted blend of everything that shared its key direction. §04 shows how the delta rule and learned gates make that blend far more selective than a plain sum; here the point is just the structural trade: fixed size, irreversible compression.

The super-block as a compression–anchor cycle

Equation (2) reads as a schedule, not a one-off layout. The $\bigotimes_{k=1}^{16}$ means the same 4-layer block is repeated 16 times: a periodic tiling, not 64 independently chosen layers. Inside each block the mixer switch $l \bmod 4$ pins three of every four layers to GatedDeltaNet and the fourth to GatedAttention, and the residual update is the one from §01: each layer adds a normalized mixer output and a normalized FFN output back into the stream. The only thing that changes between a DeltaNet layer and an attention layer is which mixer reads the stream; the FFN and the residual bus are identical.

Mechanistically, one block is a compress-then-anchor cycle. Layers 1–3 (DeltaNet) fold the stream's recent tokens into their recurrent states, integrating local context at constant memory; the fourth layer (attention) re-reads the raw embedded sequence and pulls forward anything the compression dropped, anchoring the stream against the drift that three compressive layers would otherwise accumulate. Because this 4-layer cycle repeats every four layers, drift can never build up for more than three steps before an attention layer corrects it, which is why the periodic repetition of the motif matters, not just the 3:1 count.

Stepped through, the four layers play distinct roles (the motif visualizer below labels them Compression, Refinement, Fusion, and Anchor). Layer 1 (Compression): the first DeltaNet layer writes the current token's key–value pair into a fresh recurrent state and reads back the aggregated context up to this point, so the stream now carries a compressed summary of the recent past. Layer 2 (Refinement): the next token is written into the same state and a refined summary is read back; the state is accumulating a denser representation and the stream begins to integrate mid-range dependencies. Layer 3 (Fusion): the accumulation continues, and by now the recurrent state holds a compressed view of a sizeable local window, with the stream reflecting fused local context. Layer 4 (Anchor): the Gated Attention layer sets the recurrent state aside and attends over the full raw token sequence with softmax, so any fact the compression blurred (a needle far back in the haystack) is retrieved exactly and written back into the stream. The cycle then repeats on the next block of four, with the anchor's correction carried forward as the starting point for the next three compressions.

What each layer actually does

Strip away the mixer difference and every one of the 64 layers follows the same internal recipe: the residual update from §01 made concrete. A layer takes the residual stream $\mathbf{x}_l$, normalizes a copy of it (RMSNorm), runs the mixer on that normalized copy to route information across positions, and adds the result back into the stream. It then normalizes the updated stream, runs the FFN on it to transform the representation at each position independently, and adds that back too. The mixer and the FFN divide labor cleanly: the mixer decides what each token gets to see from other tokens, and the FFN decides what each token does with what it has. The only component that varies between a DeltaNet layer and an attention layer is the mixer; the FFN and the residual bus are identical.

The FFN is on every one of the 64 layers, a 17,408-wide SwiGLU (§06), because routing information between positions is useless unless each position also has the capacity to act on it. A layer with only a mixer would move content around the stream without enriching it; a layer with only an FFN would enrich each position in isolation, unable to aggregate across tokens. Stacking both each layer lets the model alternate "gather" and "transform" 64 times, which is why depth, not width alone, drives quality. This is also why the 3:1 schedule is a statement about the mixer only: the FFN bandwidth is constant across the whole stack, and the question the backbone answers is what fraction of layers should gather cheaply (DeltaNet) versus gather exactly (attention).

One concrete consequence of mixing the two regimes is that the KV cache does not accumulate on every layer. DeltaNet layers keep their memory inside the recurrent state (a fixed tensor, not a per-token cache), so they add nothing to the sequence-length-dependent footprint. Only the 16 Gated Attention layers contribute KV cache, which means the cumulative cache grows in 16 jumps (at layers 4, 8, …, 64) rather than 64. The explorer below makes this visible: step through the layer strip and watch the running VRAM stay flat across DeltaNet layers and step up at each attention layer.

Why the 3:1 hybrid ratio is the Pareto optimum

The 3:1 ratio sits at the optimum of the memory-quality tradeoff curve. Pure linear attention eliminates the KV cache entirely but degrades on needle-in-a-haystack retrieval beyond 32k tokens. Pure softmax attention delivers perfect retrieval but demands 68.7 GB of KV cache per stream at 262k context. The 3:1 hybrid retains 99.4% of full-attention benchmark accuracy while cutting KV cache VRAM by 75%.

The tradeoff has a knee, and 3:1 sits on it. Moving from pure linear to 3:1 adds 16 attention layers, each a periodic anchor, and recovers almost all retrieval quality because the DeltaNet layers between them already handle short-range aggregation cheaply, so the anchors only need to fix long-range drift. Moving from 3:1 to 1:1 adds another 16 attention layers, but they mostly duplicate retrieval the periodic anchors already provide, so quality becomes "indistinguishable from 64A" while the KV cache doubles. Dropping to 7:1 falls past the knee: eight anchors cannot stop drift on multi-hop associative recall within a 262k-token window. The 3:1 ratio is therefore the point of diminishing returns to attention, not a guess tuned on a sweep.

The cache numbers in the table are not round figures; they fall out of the config. Each attention layer's KV cache holds a key and a value tensor for every token, sized by the number of KV heads and the head dimension; at 262k tokens in BF16 the per-layer cost is:

DeltaNet layers contribute zero to this sum, so the hybrid's cache is simply $1.07 \times n_{\text{attn}}$ GB. That is the whole story behind the "75% savings": replacing 48 of the 64 attention layers with constant-memory DeltaNet layers cuts the cache from $64 \times 1.07$ to $16 \times 1.07$ GB, a fourfold drop for a 0.6-point quality cost, and FP8 or NVFP4 KV precision (toggleable in the explorer above) scales the same product down by another 2× or 4×. The table walks this arithmetic across five mixes.

The backbone description so far treats the two mixers as black boxes, "compress" and "anchor." The next two sections open them. §04 derives Gated DeltaNet's delta rule as a gated associative memory: how a fixed-size state can write and forget content under learned gates, and why that beats a plain SSM on recall. §05 opens Gated Attention: why 24 query heads share only 4 key-value heads (GQA 6:1), how 3D MRoPE places image and text tokens on one position axis, and the output $\sigma$-gate that suppresses noise before the residual write.

Constant-memory associative recall via test-time fast weights

In 48 of the 64 layers, Qwen 3.8 replaces quadratic $T \times T$ softmax attention with Gated DeltaNet. Instead of storing past tokens in an unbounded KV cache, each layer maintains a fixed-size $128 \times 128$ associative memory matrix $S_t \in \mathbb{R}^{d_k \times d_v}$ per head, continuously updated via an error-driven delta rule with data-dependent input and decay gates.

From additive superposition to error-correcting writes

The state $S_t$ is a fast weight matrix, "fast" because it is written at inference time by the data passing through the layer, in contrast to the "slow" learned projection weights that produce the queries, keys, and values. The name DeltaNet is literally "delta rule on a net": the state is updated by an error-correcting rule rather than by accumulation. To see why that matters, start from what plain linear attention does. It writes each token additively, $S_t = S_{t-1} + v_t k_t^{\top}$. Every new key–value pair is simply summed in, so writing a new value at an address $k$ that already holds an old value does not remove the old one: the two outer products coexist, and a later query $q \approx k$ retrieves a blend of both. Over a long sequence the state saturates into a superposition of every pair ever written, and individual records become irrecoverable ghosts. That is the failure mode the widget below calls "destructive ghost traces."

The delta rule replaces the sum with an overwrite. Equation (3) writes $S_t = \alpha_t\big[S_{t-1}(I - \beta_t k_t k_t^{\top}) + \beta_t v_t k_t^{\top}\big]$. Ignore $\alpha_t$ for a moment and read the bracket in two steps. First, $S_{t-1}(I - \beta_t k_t k_t^{\top})$: because $k_t$ is L2-normalized, $k_t k_t^{\top}$ is the projector onto the $k_t$ direction, so $(I - \beta_t k_t k_t^{\top})$ removes, or scales down, whatever $S_{t-1}$ currently stores at address $k_t$. Second, $+\beta_t v_t k_t^{\top}$ stamps the new record $(k_t, v_t)$ into the vacated slot. The net effect is a targeted overwrite: writing $(k, v)$ replaces the old association at $k$ while leaving every other address intact, instead of layering the new on top of the old. With $\beta_t = 1$ this is exact ($S_t k_t = v_t$ to within the projection), which is why the delta rule is called error-correcting: the update is precisely the correction that makes the state satisfy $S k = v$.

The $\alpha_t$ out front is a global decay applied after the write, and it is what stops the fixed-size state from filling up forever; we return to it below. The structural point is that the delta rule turns the recurrent state from a lossy sum (which can only blur) into an addressable memory that can update a single slot in place. That is the mechanism behind the "constant-memory associative recall" in the section title, and it is what lets 48 of the 64 layers dispense with the KV cache entirely.

The two gates: how fast to forget, how hard to write

Equation (4) makes the two scalars in (3) data-dependent, and that data-dependence is the entire "Gated" in Gated DeltaNet. The write gate $\beta_t = 2\sigma(W_g \hat{x}_t) \in (0, 2]$ sets how hard the current token stamps its $(k, v)$ pair into the state. Its cap at 2 is a stability bound. Because $k_t$ is unit-norm, the rank-one projector $k_t k_t^{\top}$ has eigenvalue $1$ along $k_t$, so the write operator $(I - \beta_t k_t k_t^{\top})$ has eigenvalue $1 - \beta_t$ along that direction. Capping $\beta_t \le 2$ keeps $1 - \beta_t \in [-1, 1)$, a non-expanding contraction: the state norm can never grow from a write. Letting $\beta_t$ exceed $2$ would make $|1 - \beta_t| > 1$, and the write would overshoot, oscillate, and diverge over a long sequence. Within the cap, the model is free to learn $\beta_t \approx 2$ for salient tokens it must stamp firmly and $\beta_t \approx 0$ for filler it should barely write.

The decay gate $\alpha_t = \exp(-\text{softplus}(A_k \tilde{k}_t + b)) \in (0, 1)$ sets how much of the old state survives into the next step. The $\exp$ of a negative softplus is strictly below $1$ and strictly above $0$, so every step is dissipative (the state can only shrink, never amplify), and the network's job is to learn how fast to forget. Important, repeatedly-referenced tokens get $\alpha_t$ near $1$ and persist; irrelevant tokens get $\alpha_t$ near $0$ and are evicted, recycling the fixed state capacity. This is the other half of avoiding saturation: the delta rule prevents spatial collisions (overwriting the right slot), and the decay gate prevents temporal saturation (clearing stale records). Both gates are functions of the input, not fixed constants: a plain SSM decays on a fixed schedule regardless of content, which is why a content-aware gate beats it on recall.

Training parallelism: the chunked WY factorization

The recurrence in (3) is inherently sequential: $S_t$ depends on $S_{t-1}$, which depends on $S_{t-2}$, and so on back to $S_0$. During decode this is fine (one step per token, $O(1)$ memory). During pretraining, where the whole sequence must be processed at once, a naive serial scan over thousands of tokens would leave the Tensor Cores idle and dominate the step time. The fix is to chunk the sequence into blocks of size $C$ (default 64) and factor the within-chunk interaction so it can be computed as matmuls rather than a recurrence.

The factorization rests on the identity $\prod_{t=1}^{C}(I - \beta_t k_t k_t^{\top}) = I - W Y^{\top}$ (Table 3, row 5): the product of $C$ rank-one write operators inside a chunk collapses to a single $I - W Y^{\top}$, where $W$ and $Y$ are built from the chunk's keys, values, and gates. Within a chunk, all $C$ token-to-state interactions become a block-triangular $W Y^{\top}$ matmul (causal, because token $t$ only sees earlier tokens) which runs on Tensor Cores in parallel. Between chunks, only the single state matrix $S$ is carried forward sequentially, in FP32 for stability. The chunk size trades parallelism against serial depth: $C = 64$ keeps the triangular solve cheap while exposing enough parallel work to saturate the hardware. The visualizer below shows both regimes side by side.

Asymmetric head layout: 16 address channels for 48 memory heads

The DeltaNet configuration uses an asymmetric projection layout: 16 Query/Key heads ($d_k = 128$) controlling 48 Value heads ($d_v = 128$). This design optimizes memory capacity while minimizing write address collisions.

The clearest way to read this layout is as an address–content split: keys are addresses (which slot to write), values are content (what to store there). The 16 query/key heads give 16 address channels, enough routing resolution that two distinct tokens rarely hash to the same address and collide. The 48 value heads give 48 content channels, so each addressed slot stores a rich 128-vector record rather than a single scalar. The $48/16 = 3{:}1$ ratio means each address coordinates three content heads: capacity is tripled per address without tripling the address space, which keeps key routing sharp and the query/key projection small. Splitting them asymmetrically (few addresses, many content channels) is what maximizes stored information per unit of projection parameter.

• Value Head Count ($n_v = 48$): Governs the total associative memory capacity per layer ($48 \times 128 \times 128 = 786,432\text{ state floats}$). Across all 48 DeltaNet layers, the entire model maintains $37.75\text{M floats}$ ($75.5\text{ MB}$ in BF16) of recurrent state, fixed whether processing 10 tokens or 1,000,000 tokens.
• Query/Key Head Count ($n_{qk} = 16$): Each address channel coordinates writes across 3 value heads ($48/16 = 3:1$ ratio), providing sharp key routing without inflating query projection weights.
• Single Fused Input Projection: A single GEMM projects $x \in \mathbb{R}^{5120} \to \mathbb{R}^{10240}$ to simultaneously produce $Q, K, V$, gating channels, and causal depthwise 1D conv inputs in a single kernel launch.

Numerical stability checklist: the five kernel invariants

Over 262k recurrent steps, any per-step operation that expands the state even slightly will diverge. A factor of $1.001$ per step compounds to $\approx 10^{114}$ over the native context, an astronomical blow-up rather than a small drift, so the kernel is built entirely from non-expanding maps. The five invariants below are not independent precautions; they are five views of one principle: every map the state passes through must be a contraction or a norm-preservation, never an amplification.

Gated DeltaNet gives the backbone its constant-memory compression: 48 layers that integrate the stream for free, with a fixed 75.5 MB of recurrent state regardless of context length. But compression alone cannot guarantee that a specific far-back token is retrieved. The decay gate may have evicted it, and its address may have been overwritten. That guarantee is the job of the 16 full-attention layers, which skip the recurrent state and read the raw sequence directly. §05 opens that other mixer: Gated Attention, its 6:1 GQA head grouping, the 3D MRoPE that positions image and text tokens on one axis, and the output $\sigma$-gate that filters the attention output before it touches the residual stream.

Sixteen sparse anchors of exact associative recall

Every fourth layer in the 64-layer backbone (layers 4, 8, 12, ..., 64) is a Gated Attention block with Grouped-Query Attention (GQA 6:1), wide 256-dimensional query heads, partial 3D MRoPE, and a learned output sigmoid gate. These 16 layers act as global retrieval anchors that periodically refresh the recurrent associative memory.

Grouped-Query Attention: sharing the cache

The KV cache is the attention layer's memory cost, and it scales with the number of key/value heads, not the number of query heads. Standard Multi-Head Attention (MHA) gives every query head its own K and V projection ($n_{kv} = n_q = 24$), so 24 distinct key/value tensors are cached per token. At the other extreme, Multi-Query Attention (MQA) collapses all 24 query heads onto a single shared K,V pair ($n_{kv} = 1$), shrinking the cache 24× but degrading quality because every head must read the same stored representation. Grouped-Query Attention (GQA) splits the difference: query heads are partitioned into groups, and each group shares one K,V head. Qwen 3.8 uses $n_{kv} = 4$, so the 24 query heads form 4 groups of 6 (the "GQA 6:1") and the cache is 6× smaller than MHA.

Sharing the K,V heads is safe because query heads differentiate through their query projections, not through needing separate stored keys and values. Two heads reading the same K,V cache still attend to different relations, since each computes its own $Q$ and its own softmax weights; what they share is the raw key/value representation of each past token, which is largely common property. The remaining design lever is head width: Qwen 3.8 uses wide 256-dimensional heads, so each of the 4 K,V heads stores a rich 256-vector per token and each query head matches against a 256-dim key. Wide heads recover the expressivity that fewer heads sacrifice, which is why GQA-4 lands at "zero accuracy loss" in the table below while MQA does not. The σ-gate added in (7) compensates the rest.

KV-head reduction: MHA → MQA → GQA

Standard Multi-Head Attention (MHA) allocates dedicated key/value projections for all 24 query heads, requiring 103 GB of cache per stream at 262k context. Grouped-Query Attention pools 6 query heads into each KV head, reducing KV cache memory by 83.3% while preserving retrieval sharpness via wide 256-dim head projections.

The cache figures in the table follow directly from the head counts. Each token stores a key and a value for every KV head, so the per-token, per-layer footprint is $2 \cdot n_{kv} \cdot d_{\text{head}} \cdot \text{bytes}$; at BF16 with $d_{\text{head}} = 256$:

Multiplying by 262,144 tokens and 16 attention layers recovers the table's rightmost column: GQA-4 sums to $4{,}096 \times 262{,}144 \times 16 \approx 17.18$ GB (the same $1.07$ GB/layer figure derived in §03) while MHA reaches $\approx 103$ GB. The "83.3%" saving is simply $(24-4)/24$, the fraction of KV heads removed, and it costs no accuracy because the wide 256-dim heads and the σ-gate below absorb the lost expressivity.

Rotary positions: from 1D RoPE to 3D MRoPE

Before MRoPE comes RoPE itself. Rotary Position Embedding encodes position not by adding a bias but by rotating each query and key. The 256-dim head is split into 128 dim-pairs, and pair $j$ at token position $m$ is rotated by angle $m \cdot \omega_j$, where the frequencies $\omega_j = 10^{7 \cdot (-2j/d_{\text{rot}})}$ range from slow (long-wavelength, capturing distant structure) to fast (short-wavelength, capturing local order). The decisive property is that the query–key inner product after rotation depends only on the relative position $m_q - m_k$, not on either absolute position: rotating both vectors by their positions makes their dot product a function of the angle between them. RoPE extrapolates: a model trained at 32k can still attend coherently at 262k, because the relative-position geometry is unchanged.

One-dimensional RoPE has a single position index $m$, which is fine for text but breaks for images and video. A pixel at row $h$, column $w$ in frame $t$ does not have a single natural 1D position; forcing it onto one flattens the 2D/3D structure the model needs to reason about. Multimodal RoPE (MRoPE) solves this by splitting the rotary dim-pairs into three disjoint sections and rotating each section by an independent position axis. The config mrope_section: [11, 11, 10] assigns 11 dim-pairs to the temporal axis $t$, 11 to the height axis $h$, and 10 to the width axis $w$: 32 pairs in total, i.e. 64 of the 256 head dimensions, matching the partial rotary ratio of 0.25.

The three axes cooperate to place any modality on one position space. A text token has a temporal position and a trivial spatial one, so it rotates mainly along the $t$ section. An image patch has a row and column, so it rotates along the $h$ and $w$ sections with $t$ held fixed. A video frame's patches rotate along all three. Because the sections are disjoint, the axes never interfere: the $h$-rotation of an image patch is identical whether the patch is in frame 1 or frame 100, which is what lets one position embedding serve text, images, and video without retraining. The remaining 192 dims stay position-independent, so content similarity is not entirely overwritten by positional structure. The dial below shows the per-pair rotation this produces.

The output σ-gate: filtering before the residual write

Equation (7) is the "Gated" in Gated Attention, and it sits between the softmax retrieval and the residual stream. The attention output $\text{Attn}(Q,K,V)$ is element-wise multiplied by $\sigma(x W_g)$ before the output projection $W_o$ adds it back into the stream. The gate is a per-channel scalar in $(0,1)$ computed from the layer's input $x$, so it is data-dependent, not a fixed mask. When the retrieved context is relevant, the gate opens toward $1$ and the attention output passes through to the residual; when the retrieved context is irrelevant or noisy, the gate falls toward $0$ and the attention contribution is suppressed before it can pollute the stream.

This is the mechanism that lets the model run only 4 KV heads without losing quality. Aggressive KV-head reduction (GQA-4) saves the cache but means each retrieval reads a coarser shared representation, which can surface loose matches; the σ-gate discards those loose matches at the output rather than letting them propagate. In effect the gate gives each attention layer a learned on/off switch for its own contribution: a layer that finds nothing useful for the current token can near-silence itself, contributing almost nothing to the residual, while a layer that retrieves a key fact passes it through at full strength. That is the "compensated by σ-gate" note in the GQA table, and it is what closes the quality gap to full MHA at a sixth of the cache cost.

The 16 Gated Attention layers are the backbone's exact-retrieval path: GQA-4 keeps their cache servable at 17.18 GB, MRoPE lets them position text and vision on one axis, and the σ-gate ensures their contributions are signal, not noise. But attention only routes information between positions. It does not change what a position can represent. That transformation is the job of the FFN, which sits in every one of the 64 layers and is the subject of §06: the 17,408-wide SwiGLU that holds the majority of the model's parameters, and the Pre-RMSNorm residual bus that the mixer and FFN both write into.

Where 62% of parameters and 71% of arithmetic reside

Every one of the 64 hybrid layers terminates with an identical SwiGLU Feed-Forward Network (FFN) of intermediate width $d_{ff} = 17,408$. Carrying 17.11 billion parameters, the FFN blocks form the primary factual associative storage of the network, reading from and writing to a 5,120-dimensional pre-norm residual communication bus.

The FFN as a per-position key–value memory

The labor division established in §03 is exact: the mixer (DeltaNet or attention) moves information across positions and never looks at two tokens through the same matrix; the FFN transforms each position independently, with no knowledge of its neighbours. A 5,120-vector $\mathbf{x}_l$ arrives at the FFN, and a new 5,120-vector leaves, one token at a time, through three matrices that are shared across all positions but applied per-token. Where the mixer answers "what does this token know about?", the FFN answers "what does this token mean, given what it now carries?"

The clearest way to read eq.(8) is not as a nonlinear MLP but as an associative memory in the sense of Geva et al. Reshape the up-projection as $17,408$ row-vectors $\mathbf{k}_i$ (the $i$-th row of $W_{\text{up}}$) and the down-projection as $17,408$ column-vectors $\mathbf{v}_i$ (the $i$-th column of $W_{\text{down}}$). Each hidden unit $i$ computes a scalar address $a_i = \mathbf{x} \cdot \mathbf{k}_i$, passes it through an activation, and votes $\mathbf{v}_i$ into the output:

The FFN, not the mixer, holds the network's factual storage: each $(\mathbf{k}_i, \mathbf{v}_i)$ pair is a memory slot that fires when the incoming residual stream matches its key, contributing its value to the bus. With $17,408$ slots per layer and $64$ layers, that is $1.11$ million independent memory records, the substrate on which facts are stored during pretraining and recalled at inference. The mixer's job is to deliver the right $\mathbf{x}$ to the right layer so the right slots fire.

The intermediate width $d_{ff} = 17,408 \approx 3.4 \cdot d_{\text{model}}$ is the network's single largest capacity lever. Unlike head counts or layer depth, it scales storage and per-token compute linearly, and it is where the bulk of the 27.58 B parameters actually live. The classic transformer uses a $4\times$ expansion; Qwen 3.8 uses $3.4\times$ because the SwiGLU gate recovers enough expressivity per unit that a smaller multiple suffices, as the next subsection shows, saving parameters that are instead spent on the 64-layer depth and the 5,120-wide bus.

Why SwiGLU: a gate per unit kills the dead-unit problem

To see what eq.(8) buys over a plain activation, walk down the ladder of FFN transfer functions that Fig. 3 plots. A ReLU unit fires as $\max(0, \mathbf{x}\cdot\mathbf{k}_i)$: the unit is either fully on or fully off, and once its pre-activation $\mathbf{x}\cdot\mathbf{k}_i$ is pushed negative by gradient updates, it receives zero gradient and dies permanently, a wasted memory slot. GELU and SiLU ($z\sigma(z)$) soften this: the transition through zero is smooth, so a "barely off" unit still leaks a small gradient and can recover. But in all three, the on/off decision is a fixed function of the same dot product that produces the value: the unit cannot be silenced for one input and active for another that happens to land at the same address.

The Gated Linear Unit (GLU) breaks this coupling by introducing a second projection, the gate. Instead of $\phi(\mathbf{x}\cdot\mathbf{k}_i)\,\mathbf{v}_i$ with a fixed $\phi$, SwiGLU computes

Now the on/off decision and the value magnitude come from different directions in the 5,120-dim input space: the gate vector $\mathbf{g}_i$ is learned independently of the key $\mathbf{k}_i$. A unit can be wide-open for inputs that match $\mathbf{k}_i$ but gated off for inputs that match $\mathbf{g}_i$ in a conflicting direction: a data-dependent, second-order filter that no fixed activation can express. Because the gate is its own $5120 \times 17408$ matrix, this expressivity costs one extra projection (the third in eq.(8)), which is the price SwiGLU pays and the reason the expansion ratio can drop from $4\times$ to $3.4\times$ without losing capacity.

Two consequences fall out of this gating. First, dead units largely disappear: even when $\mathbf{x}\cdot\mathbf{k}_i < 0$ would silence a ReLU slot, the gate $\mathbf{x}\cdot\mathbf{g}_i$ can still route a non-zero contribution, so the gradient through $\mathbf{k}_i$ and $\mathbf{g}_i$ rarely vanishes simultaneously. Second, the per-unit output is now a product of two input-dependent scalars, so the FFN can represent multiplicative interactions between features, something a plain MLP can only approximate wastefully through its width. The activation explorer below sweeps $x$ and shows the resulting non-monotonic, self-gated transfer surface.

The 17.11 billion: where the parameters actually live

Before the FLOP table, account for the parameters themselves. Every figure in Table 5 falls out of one multiplication. Each FFN holds three matrices: the gate $W_{\text{gate}}$, the up $W_{\text{up}}$, and the down $W_{\text{down}}$. The first two lift the 5,120-dim bus up to the 17,408-wide hidden store; the third projects it back down. In BF16, that is

Three observations worth holding onto. (i) The extra gate matrix is the SwiGLU tax: a vanilla 2-matrix FFN of the same width would cost $178.3$M/layer; SwiGLU's third projection adds $89.1$M, and in return buys the per-unit gating that lets the width shrink from $4\times$ to $3.4\times$, a net parameter win. (ii) The width $17,408$ is rounded to a multiple of $256$ for the same Tensor-Core MMA-tile reason as the vocabulary padding in §02: a $256$-wide tile keeps the down-projection's GEMM dimensions clean on Hopper/Blackwell, so the "wasted" $17,408 - 17,408$-ish slots are not waste but alignment headroom. (iii) Because the FFN is the only sub-layer present on all 64 layers (the mixers split 48/16), its per-layer parameter mass dominates the total: $267$M $\times$ $64$ swamps the mixers' $83.9$M $\times$ $48$ and $104.9$M $\times$ $16$ combined, which is the structural reason the FFN is both the factual store and the arithmetic center.

Arithmetic FLOP accounting across the forward pass

Per token, per layer, matrix multiplication FLOPs equal $2 \times \text{parameters}$. In Qwen 3.8, arithmetic is heavily dominated by FFN tensor-core operations:

RMSNorm bus arbitration & QK-normalization

Pre-RMSNorm: why the bus stays at variance 1

Eq.(10) is the bus from §01 made concrete, and its pre-norm placement is the single most load-bearing micro-decision in the architecture. Read the two clauses carefully: $\text{RMSNorm}_1$ and $\text{RMSNorm}_2$ sit inside the mixer and FFN arguments, not outside. The sub-layers never see the raw residual $\mathbf{x}_l$; they see a normalized view of it, while the un-normalized $\mathbf{x}_l$ is the thing that accumulates.

Contrast this with post-norm, $\mathbf{x}_{l+1} = \text{RMSNorm}(\mathbf{x}_l + \text{Mixer}(\mathbf{x}_l))$. There the norm applied after the residual add rescales the whole stream each layer, so early-layer signal gets progressively shrunk and gradients through the deep stack become unstable. That is the divergence the "Unnormalized Stream" toggle in the variance tracker above dramatizes, but post-norm has its own failure mode where the norm fights the residual. Pre-norm separates the two concerns: the residual stream is free to grow (it is a sum, not a normalised quantity), while every sub-layer reads a unit-scale input and therefore produces a unit-scale output regardless of depth. Each layer contributes a depth-controlled increment; the bus is a running sum of bounded updates rather than a recursively re-scaled vector.

The arithmetic behind the widget's stable line is direct. RMSNorm divides by the root-mean-square, so the normalized vector has RMS exactly $1$ (before the learned gain $\boldsymbol{\gamma}$). If a sub-layer's output magnitude is roughly independent of its input magnitude (true for both the $\sigma$-gated attention and the gated SwiGLU, whose gates re-normalize large inputs), then each residual update adds a term of bounded variance, and the stream's variance grows roughly linearly with depth, not exponentially:

The tracker shows the unnormalized stream blowing past usable dynamic range around layer 20, since $c^{20}$ for even a modest per-layer gain $c > 1$ is already catastrophic, while the pre-norm line holds at $\text{var} \approx 1$ across all 64. The learned per-channel gain $\boldsymbol{\gamma} \in \mathbb{R}^{5120}$ is the one knob RMSNorm keeps from full LayerNorm: it lets each channel re-scale after normalization, recovering any deliberate anisotropy (some channels should carry more signal than others) without re-introducing the depth instability. RMSNorm drops LayerNorm's mean subtraction because the residual stream is approximately zero-mean by symmetry of the sub-layer outputs, so centering adds compute and a stat that is nearly constant, a small but real saving repeated $129$ times.

The bus is the same 5,120-dimensional channel introduced in §01; here it acquires its traffic rules. Each of the 64 layers is a pair of parallel readers of the normalized stream: the mixer reads $\text{RMSNorm}_1(\mathbf{x}_l)$ and routes across positions, the FFN reads $\text{RMSNorm}_2(\mathbf{x}_l)$ and transforms per-position, and both write back additively. Because both readers see the same normalized input and write to the same bus, the mixer and FFN never contend: the mixer decides which positions' vectors arrive at the FFN's slots, the FFN decides what those slots return to the bus, and the residual sum composes the two without interference. The 129 RMSNorms are the arbiters that keep this traffic at unit scale; QK-Norm does the same job inside attention, holding the $256$-dim dot-product logits in a bounded softmax regime so the §05 $\sigma$-gate sees well-scaled probabilities rather than saturated ones.

There are exactly 129 RMSNorms in the core backbone: two per layer (pre-mixer and pre-FFN across 64 layers = 128) plus the final pre-LM head norm. QK-Norm inside all 16 attention layers also keeps dot-product logits $(q^T k / \sqrt{d_k})$ inside a bounded softmax regime, preventing gradient explosions during large-batch pretraining. With the bus stabilised and the FFN's 17.11 B parameters seated as the factual store, the remaining question is how non-text modality enters that bus, which is the vision front-end of §07.

How continuous pixels become discrete 5,120-dimensional tokens

Qwen 3.8 is natively multimodal from the ground up: a SigLIP-style Vision Transformer (ViT) converts images and high-fps video frames into dense patch embeddings, a 2-layer MLP projector aligns visual geometry with the language model's 5,120-dim residual stream, and 3D MRoPE assigns spatial-temporal coordinates $[T, H, W]$. Roughly 333 weight tensors (~0.9B parameters) comprise the visual front-end.

The five-stage bridge, from continuous pixels to the residual bus

The backbone built in §§03–06 is a discrete-token machine: it understands only one currency, a 5,120-dimensional vector on the residual bus, addressed by the FFN's 1.11 M key–value slots and routed by the §05 attention. An image, by contrast, is a continuous two-dimensional field of RGB triples: it has no tokens, no position indices, and no native address on that bus. The vision front-end is the translator that converts one into the other, and its five stages are best read as a single contraction: pixels → patches → patch features → merged features → bus-native tokens, with dimension changing $3 \to 1{,}536 \to 1{,}536 \to 6{,}144 \to 5{,}120$ and token count falling by $4\times$ at the merge.

Patchify turns a grid of pixels into a sequence of patch tokens, the vision analogue of §02's token-embedding lookup, lifting raw input into the first feature space. SigLIP ViT runs the same self-attention machinery as §05 over those patch tokens, producing features that carry semantics rather than raw color. 2×2 Merge compresses four neighbouring patch features into one, trading sub-pixel acuity for a $4\times$ token-count cut. MLP Project is the single dimension-changing step, the only place the 1,536-dim vision space is mapped onto the 5,120-dim bus. MRoPE Splice attaches the 3D $[T,H,W]$ coordinates from §05 and interleaves the result into the text token stream. After stage 5, a vision token is, to the backbone, indistinguishable from a text token: same bus, same FFN slots, same attention, same MRoPE. That equivalence is the whole point of a native multimodal design.

The widget below lets you click through the five stages; the prose that follows unpacks the mechanism behind each one, beginning with the patchify step that every later stage depends on.

Patchify: a strided convolution is a per-patch linear projection

Stage 1 is a single convolutional layer with a $14 \times 14$ kernel, stride $14$, lifting $3$ input channels to $1{,}536$ output channels. Read it mechanically: the kernel slides over the image in non-overlapping $14 \times 14$ blocks, and at each placement it takes the $14 \cdot 14 \cdot 3 = 588$ pixel values inside the window, flattens them, and multiplies by a $588 \to 1{,}536$ weight matrix. That is, point for point, a linear projection of one image patch into a $1{,}536$-dim embedding vector, implemented as a convolution purely because a conv expresses "apply this same projection to every spatial window" in one fused Tensor-Core kernel. A $224 \times 224$ tile thus yields $(224/14)^2 = 16^2 = 256$ patch tokens, each a $1{,}536$-vector. Higher resolutions are handled by tiling the image into $224$-px blocks and processing each tile, then concatenating. That native-resolution approach lets a 4K diagram produce tens of thousands of patches without retraining the ViT at that scale.

The patchify step is the direct vision analogue of §02's token embedding. There, a discrete token id $x_t \in \{0, \dots, 248319\}$ indexes a row of a $248320 \times 5120$ embedding table, a lookup. Here, a continuous $588$-vector patch is multiplied by a $588 \times 1536$ matrix, a projection. Both lift raw input into the first feature space their respective towers consume; the only difference is that text input is discrete (so a lookup suffices) and image input is continuous (so a linear map is required). After patchify, the image is no longer pixels. It is a sequence of $256$ tokens on a $1{,}536$-dim channel, ready for the same transformer machinery that processes text.

The SigLIP ViT: attention over patches, the same recipe as §05

Once patchify has produced a sequence of $1{,}536$-dim tokens, the ViT tower encodes them with $30$ layers of self-attention: $16$ heads, $d_v = 1{,}536$, MLP width $6{,}144$ ($4\times$ expansion, the classic ratio, since the ViT predates the SwiGLU parameter-saving of §06). The mechanism is identical to §05's Gated Attention: query/key/value projections, softmax over patch positions, a residual bus, Pre-RMSNorm. The only differences are the working dimension ($1{,}536$ not $5{,}120$), the position system (learned 2D patch-position embeddings rather than MRoPE, since within the ViT every token is an image patch with a fixed grid coordinate), and the absence of the GQA/MRoPE machinery that the multimodal bus requires. In other words, vision and text use the same mixer once input has been lifted into token space; the front-end's entire job is to get pixels into that space and then align it to the LLM's bus.

The ViT's $849.6$ M parameters, the overwhelming bulk of the $0.9$ B visual front-end, are spent exactly where §06 predicted they would be: in the per-layer FFN (here a $4\times$-expanded MLP, $30$ layers of $1536 \times 6144$ projections), with the attention projections a small minority. The same FFN-as-associative-memory reading applies: the ViT's slots store visual primitives (edges, textures, object parts, OCR glyphs) that fire when a patch's features match their keys. By layer 30 the output is a sequence of semantic feature vectors, one per patch, that no longer resemble pixels.

Sigmoid contrastive pretraining (the "S" in SigLIP)

Standard CLIP models train vision-text pairs with a global softmax normalization across batch size $B$. This creates a severe synchronization bottleneck in multi-GPU distributed clusters because every negative sample competes in a shared denominator. SigLIP replaces global softmax with independent pairwise sigmoid binary classification losses:

The mechanism is easier to see by contrasting the two losses term by term. CLIP's InfoNCE loss for pair $i$ is $-\log \frac{\exp(t_i^T v_i / \tau)}{\sum_{j=1}^B \exp(t_i^T v_j / \tau)}$: a softmax whose denominator sums over every other pair $j$ in the batch. That denominator is the bottleneck: to compute it correctly, every GPU must contribute its local pairs to a global allgather, so batch size $B$ is bounded by inter-GPU communication, not by how much data you have. SigLIP replaces the single $B$-way softmax with $B^2$ independent binary decisions: for each $(i, j)$ pair, compute $z_{ij} = t_i^T v_j / \tau - b$ and apply a sigmoid, treating matching pairs $(i=j)$ as positives (target $+1$) and mismatched pairs $(i \ne j)$ as negatives (target $-1$):

Because each $\ell_{ij}$ depends only on pair $(i,j)$ and not on any global sum, no allgather is required. The loss decomposes across GPUs, and batch size can grow to $32{,}000+$ pairs per step limited only by device memory. The learned bias $b$ shifts the decision boundary so the sigmoid is not pinned at $0.5$ when cosines are near zero, and the learned temperature $\tau$ controls how sharply the similarity comparison gates. A second, subtler benefit: in CLIP each negative is only a fractional pressure inside one softmax, so a single hard negative barely registers; in SigLIP each negative is its own full-strength binary decision, so fine-grained distinctions (a zebra vs a horse, an uppercase O vs a zero) get a complete gradient signal, which is the mechanism behind the richer OCR and spatial semantics.

SigLIP therefore scales to global batches of 32k+ pairs per step, yielding visual representations with significantly richer spatial and fine-grained OCR semantics than CLIP.

The 2×2 spatial merge & MLP projector

Spatial merging trades sub-pixel acuity for a $4\times$ reduction in sequence length, transforming a 10,764-token 1080p frame into a 2,691-token payload that the hybrid backbone ingests at thousands of tokens per second.

Read the two projector lines as a single dimensional handoff. The merger concatenates four neighbouring $1{,}536$-vectors into a $6{,}144$-vector. This is not a learned operation, just a spatial reshape that says "treat a $2 \times 2$ block of patches as one super-patch." The MLP projector then does the only dimension change in the whole front-end: a SwiGLU-style 2-layer MLP $6{,}144 \to 6{,}144 \to 5{,}120$, landing the merged feature exactly on the residual bus from §06. After this projection the vision token is bus-native: the backbone's attention will route it, its FFN slots will fire on it, and the §05 MRoPE will position it, exactly as if it were a text token. That is why the projector, at $69.2$ M parameters, is tiny compared with the ViT but structurally the most important component in the tower: it is the bridge from the vision's $1{,}536$-dim channel onto the LLM's $5{,}120$-dim bus, and the place where "vision feature" becomes "token the backbone can reason about."

The token-budget arithmetic the calculator above runs is one multiplication of patchify and one division of merge. Raw patch tokens for an input are $(W/14) \times (H/14) \times T_{\text{frames}}$; the $2 \times 2$ merge then divides by $4$ (modulo tile-boundary rounding):

Each of those tokens costs the same context budget as a text token, a $5{,}120$-dim slot in the $262{,}144$-token window, so a single 1080p screenshot consumes about $2.7$ k of context, a 4K diagram about $10.7$ k, and a 64-frame video proportionally more. This is the resolution-versus-context tradeoff the calculator makes explicit: the front-end can ingest arbitrarily high resolution, but every $14 \times 14$ patch is eventually paid for in the backbone's attention FLOPs and KV cache, so the $2 \times 2$ merge is the lever that keeps rich visual input from crowding out the text reasoning around it.

The final stage, MRoPE splice, attaches coordinates and interleaves. Image patches receive a $[T, H, W]$ triple with $T = 0$ and $(H, W)$ set to their grid position; video frames vary $T$ by frame index. This is precisely what §05's MRoPE was built for: its three disjoint $[11,11,10]$ rotary sections give vision tokens independent $H$ and $W$ axes that text tokens leave at zero, so the two modalities share one position space without collision. The splice then drops the vision tokens into the text stream at the position of the <image> / <video> sentinel, after which the backbone has a single, mixed-modality sequence to route and transform.

Visual tower component accounting

The $0.90 + 849.6 + 69.2 = 919.7\text{M} \approx 0.9\text{B}$ accounting has one structural footnote: the visual front-end is conditionally active. Unlike the backbone, which fires on every token, these $333$ weight tensors only execute when the input contains an <image> or <video> sentinel; for a pure-text request they sit idle in HBM, costing VRAM but zero FLOPs. That is what makes a $0.9$ B vision tower affordable on a model whose FFN alone is $17.11$ B: it is a detachable peripheral, not a always-on component, and the §12 inference optimizations can keep it resident or swap it depending on whether the request is multimodal. The remaining structural question is how the backbone is trained to look ahead more than one token at a time, which the Multi-Token Prediction heads of §08 answer.

Dual-purpose multi-future supervision and zero-overhead drafting

Qwen 3.8 trains with native Multi-Token Prediction (MTP) heads integrated directly into the backbone. During pretraining, MTP provides dense multi-step credit assignment that deepens associative memory retention across DeltaNet layers; at inference, those same heads emit candidate draft tokens for free, accelerating autoregressive generation by 1.8×–2.4× without requiring a secondary draft model.

Why look ahead? The one-token horizon problem

A purely next-token language model, eq.(12) alone, has a structural mismatch between how it is trained and how it is used. At training time the backbone's hidden state $\mathbf{h}_t$ receives a gradient signal from exactly one loss term, the prediction of $x_{t+1}$. It is therefore optimised to encode "what comes next," a one-token horizon. At inference time, however, that same $\mathbf{h}_t$ must support a long coherent continuation: every token from $t+1$ onward is generated by re-reading the growing context, but the quality of those generations is bounded by how much lookahead $\mathbf{h}_t$ was pressured to encode during training. The model is never explicitly trained to plan beyond one step.

MTP closes that gap by supervising $\mathbf{h}_t$ against several futures at once. With $K=2$ auxiliary heads, the same hidden state is asked to predict $x_{t+1}$ (the main head), $x_{t+2}$ (head 1), and $x_{t+3}$ (head 2) simultaneously. The gradient now flows back from three targets through three heads, so $\mathbf{h}_t$ is pressured to carry information useful for a two-token lookahead rather than a one-token guess. For the 48 DeltaNet layers this is especially load-bearing: their recurrent state $S_t$ (§04) is an associative memory that must remain retrievable for $K$ future positions, so the multi-future loss forces $S_t$ to store durable, multi-step structure rather than a single ephemeral prediction. The benefit is denser supervision, shorter gradient paths, and a state trained to age well.

The multi-future training objective

Standard autoregressive language models optimize only next-token cross-entropy $L_{\text{LM}}$. MTP augments this with $K$ auxiliary prediction heads ($K=2$ in Qwen 3.8), each predicting future tokens $x_{t+1+k}$ conditioned on the backbone's final hidden state $h_t$ and embedded prior drafts:

This yields two benefits: (1) Dense Credit Assignment: Backpropagation flows simultaneously from $x_{t+1}, x_{t+2}, x_{t+3}$, shortening gradient path lengths and forcing the 48 DeltaNet layers to maintain associative states $S_t$ that remain valid across multi-token lookaheads. (2) Zero-Overhead Speculation: At runtime, the depth-1 MTP heads generate candidate tokens in the same forward step as the main model, completely eliminating the parameter and VRAM overhead of separate draft models.

The cascade: how $K$ heads share one hidden state

The $K$ MTP heads are not $K$ independent predictors; they form a serial cascade that reuses the backbone's $\mathbf{h}_t$ and the shared $248{,}320$-token embedding. The main LM head produces $P(x_{t+1} \mid \mathbf{h}_t)$. Head $k$ produces $P(x_{t+1+k} \mid \mathbf{h}_t, \tilde{x}_{t+1}, \dots, \tilde{x}_{t+k})$. It is conditioned on the embeddings of the tokens already drafted by the shallower heads, which is what the $\tilde{x}$ symbols in eq.(13) denote. During training these drafts are teacher-forced to the ground-truth tokens, so each head learns the conditional distribution of its future token given the true prefix; during inference they are the argmax outputs of the previous head, chained forward. Because every head taps the same $\mathbf{h}_t$ and the same embedding table, the only new parameters are the $K$ projection matrices, the $+2.4\%$ per head that the telemetry card reports ($+4.8\% \approx 1.32$ B for $K=2$, $\sim 2.6$ GB in BF16), with zero extra vocabulary or embedding cost.

The draft–verify–accept cycle, step by step

At inference the MTP heads convert one forward pass into $K+1$ candidate tokens, which the target model then verifies in a single parallel pass. The cycle is:

(1) Draft. A forward pass over the current prefix produces $\mathbf{h}_t$. The main head emits the root token $x_{t+1}$; MTP head 1, reading $\mathbf{h}_t$ and the embedding of $x_{t+1}$, emits draft $\tilde{x}_{t+2}$; head 2, reading $\mathbf{h}_t$ and the embeddings of $x_{t+1}, \tilde{x}_{t+2}$, emits draft $\tilde{x}_{t+3}$. The $K+1$ tokens are appended to the sequence as a speculative extension.

(2) Verify. One forward pass over the $K+1$-token extension scores all positions in parallel, under a 2D tree causal mask (Fig. 5b) that lets each position attend only to its causal ancestors and not to speculative siblings. The target model thus produces its own argmax at each of the $K+1$ positions in the time it takes to run one pass, not $K+1$ sequential passes.

(3) Accept-and-resample. Walk the extension left to right and accept the longest prefix where the draft's token equals the target's argmax. At the first divergence, discard the remainder and resample that single position from the target's distribution. Advance the context by (accepted tokens $+ 1$), and repeat.

The speedup lives in step 2. Autoregressive generation is memory-bandwidth-bound: the dominant cost per token is reading the $1.27$ B-parameter untied LM head from HBM to produce one distribution (the §01 "MTP amortizes LM-head bandwidth" note). By verifying $K+1$ tokens in one pass, MTP amortises that HBM read across $K+1$ candidates, so the bandwidth-bound head, the inference bottleneck identified in Table 5, is paid once per $K+1$ tokens rather than once per token. The verification pass itself is compute-heavier (it scores $K+1$ positions), but on memory-bound consumer GPUs the bandwidth saving dominates, which is why the wall-clock gain is $1.8$–$2.4\times$ rather than the naive $K+1$.

Branching speculative decoding needs a tree mask. A linear chain mask treats the drafts as one serial continuation; a branching tree mask (beam $2$ in Fig. 5b) lets each head propose multiple alternative tokens, arranged so position $i$ attends only to its ancestors. The mask matrix $M_{i,j} = 0$ if $j$ is an ancestor of $i$ and $-\infty$ otherwise, an asymmetric triangular block structure that evaluates every branch of the speculative tree concurrently in a single self-attention computation, with no cross-pollination between sibling hypotheses. The widget below lets you toggle between the linear chain and the branching tree to see how the mask restructures.

Acceptance-rate mathematics & exact distribution preservation

Speculative decoding carries one mathematical guarantee: the output distribution is provably identical to standard greedy or nucleus autoregressive sampling from the target model. Draft proposals can only accelerate generation; they can never alter the output probabilities or introduce draft-model hallucinations:

For $K=2$ draft heads at $\beta = 0.85$, the model accepts an expected $\mathbf{2.57\text{ tokens per forward pass}}$, translating to a net $>2.2\times$ wall-clock throughput increase on consumer GPUs where memory bandwidth is the primary bottleneck.

The $\frac{1 - \beta^{K+1}}{1 - \beta}$ expression is a finite geometric series, and it falls straight out of the accept-and-resample rule. The root token $x_{t+1}$ is the main head's own output, so it is always correct. That is the leading $1$. Draft $\tilde{x}_{t+2}$ is accepted with probability $\beta$ (the chance the draft matches the target's argmax), contributing $\beta$. Draft $\tilde{x}_{t+3}$ is accepted only if both it and the draft before it match, contributing $\beta^2$, and so on. Summing the expected contributions of the root and the $K$ drafts gives

Plugging the three domain presets (with $K=2$) recovers the simulator's speedup tiers exactly: structured code ($\beta = 0.88$) gives $(1 - 0.88^3)/0.12 = 2.66 \to \sim 2.3\times$ after the verification-pass overhead; standard code ($\beta = 0.78$) gives $2.39 \to \sim 1.9\times$; creative text ($\beta = 0.55$) gives $1.85 \to \sim 1.4\times$. The $\gamma_{\text{verify}}$ term in the speedup factor is the wall-clock cost of the single verification pass relative to a generation step, small on bandwidth-bound hardware, which is why $\mathcal{S}$ tracks $\mathbb{E}[\text{tokens}]$ closely there and sags on compute-bound servers.

The "exact distribution preservation" guarantee, the $\text{KL} = 0.0$ in the telemetry card, is a consequence of step 3, not a separate mechanism. Accepted drafts are, by definition, tokens where the draft's choice equals the target's argmax, so they are samples the target would have produced anyway. At the first divergence the correction token is resampled from the target's full distribution, not the draft's, so the marginal distribution of every emitted token is exactly the target's $P(x_{t+1} \mid x_{\le t})$. Drafts can only accelerate; they can never inject their own probabilities, which is why a wrong draft is silently discarded rather than corrupting the output. That is the Leviathan et al. rejection-sampling guarantee that makes speculative decoding distribution-safe.

Native MTP vs separate draft model comparison

Each row of Table 7 is a consequence of the MTP heads living inside the backbone rather than beside it. The shared $248{,}320$-vocabulary LM head (§02) means drafts are produced in the exact token space the target reasons in, with no tokenizer drift and no embedding misalignment, which is the single largest source of acceptance-rate collapse in separate-draft setups. The shared KV cache means the verification pass reuses the backbone's already-materialised keys and values rather than maintaining a second cache that must be synchronised token-by-token. And the training-regulariser row is the asymmetry that makes MTP structurally superior: a separate draft model is a pure inference tax with zero benefit to the foundation model, whereas the MTP multi-step loss is paid during pretraining for a backbone that is more capable at single-token generation too. The dense credit assignment of §08's first subsection makes the base model better, and the drafting capability is a free by-product. The cost is the $+4.8\%$ parameter overhead, carried only at inference and shed when MTP is disabled for non-speculative serving.

With the architecture's forward path, multimodal ingress, and training-time lookahead all specified, the remaining input to the model is the one no amount of architecture can compensate for: the data it is trained on. The §09 pipeline turns $120$ PB of raw web into the $36$ T tokens that fill the FFN's memory slots and the MTP heads' futures.

From 120 petabytes of raw web to 36 trillion clean tokens

A 27.6B-parameter hybrid model demands high-quality data at scale: 36 trillion tokens spanning 119 natural and programming languages, with ≈39% synthetically amplified through automated verifier loops. This section details the ingestion funnel, deduplication mathematics, model-in-the-loop synthesis, and sequence packing protocols.

The funnel: 120 PB of raw web, distilled to 36 T tokens

Read the pipeline as a chain of lossy filters, each removing one specific failure mode. The numbers in the widget above are not arbitrary volumes but the survivors of the previous stage's filter, and the retention ratios tell you where the data actually comes from:

Stage 1 harvests $120$ PB of raw web (Common Crawl, books, code repositories, PDFs, and image sets), most of it duplicate, spam, SEO boilerplate, or non-text. Stage 2 strips HTML markup and runs OCR over images and PDFs to extract $45$ PB of parsed text (a $37.5\%$ retention; the discarded bytes are image data and markup, the latter routed separately to the §07 vision tower). Stage 3 is the highest-leverage cut: deduplication, quality classification, language identification, and decontamination reduce $45$ PB of parsed text to $22$ T organic tokens, the human/web portion. Stage 4 adds $14$ T of verifier-gated synthetic data (Table 8) to fill capability gaps the organic corpus under-covers, and Stage 5 packs the $36$ T total into fixed-length training shards. The $39\%$ synthetic share is not a budget target but the output of a generate-versus-verify tradeoff: synthesis is cheap to produce and expensive to verify, so $14$ T is where the verification budget saturated.

Why deduplication is the highest-leverage filter

The single largest waste in a raw web crawl is repetition: the same article, boilerplate footer, or license text is mirrored across thousands of sites, so a naive corpus is dominated by near-identical copies. Training on duplicates is doubly harmful. It spends FLOPs re-learning what the model already knows, and it biases the §06 FFN's key–value memory toward the duplicated content, since the slots that fire on boilerplate get reinforced out of proportion and crowd out rarer, more informative facts. Deduplication removes both failures at once, and because it operates on cheap string signatures rather than expensive model scores, it is the most FLOPs-efficient cut in the funnel: every duplicate token removed is a training FLOP that never runs.

The mechanism is MinHash locality-sensitive hashing. Each document is shingled into overlapping $n$-gram windows, each shingle is hashed, and the document is represented by the minimum hashes of $b$ random bands. Two documents sharing a band are near-duplicate candidates; exact-match on the band signatures clusters them without ever comparing documents pairwise (which would be $O(N^2)$ over billions of documents). Near-dup removal also catches template boilerplate, the pages that share $80\%$ of their content with a million others, which is the bulk of what Stage 3 discards. Running dedup before the quality classifier matters: it is cheaper to deduplicate cheap data than to first pay a model to score copies you are about to throw away.

The curriculum mix is a schedule, not a ratio

The domain bar above is an annealed schedule, not a fixed recipe. It shifts across the three pretraining stages, and the shift is the data-side analogue of the MTP $\lambda_k$ annealing in §08. Stage 1 ($30$ T) is broad world knowledge: web and books dominate, because the model must first fill the FFN's memory slots with the long tail of facts, entities, and common-sense associations that later stages will reason over. Stage 2 ($+5$ T) tilts hard toward STEM and code, so the bar's code and STEM segments swell, because once the slots exist, the model needs formal-reasoning distributions to learn the multi-step inference (and the §08 multi-future loss) that the organic web under-represents. Stage 3 ($\sim 1$ T) shifts to long-horizon $262$k-token documents, up-weighting sequences that exercise the §03 DeltaNet recurrent state and the §05 262k MRoPE extrapolation at their full length.

The principle is the same one that governs §10's loss-temperature annealing: the easiest, broadest distribution is learned first, and the harder, more specialised distributions are introduced only once the base representation is stable. A fixed mix from step one would either starve the reasoning capabilities (if web-heavy) or destabilise early training (if code-heavy before the base facts exist). The schedule lets each capability arrive when the model has the substrate to absorb it, which is why the widget's three stage buttons show three genuinely different bars, not three views of one corpus.

Sequence packing: the FLOPs-per-byte lever

Documents have a power-law length distribution: most are short, a few are very long, so naively padding each to a fixed training length wastes roughly half the compute on zeros. Sequence packing eliminates that waste: variable-length documents are sorted and greedily bin-packed into fixed-length sequences (4,096 for the dense pretraining stages, $262{,}144$ for the long-context stage), and best-fit-decreasing packing achieves the $\approx 98\%$ fill rate the widget reports. The arithmetic of the saving is direct: a $98\%$ fill versus a $\sim 50\%$ padded fill is nearly a $2\times$ increase in useful tokens per forward pass, which is a $2\times$ throughput gain at zero model cost, purely from how the data is laid out.

Packing introduces one hazard that the widget's two modes dramatise. When documents A and B share a packed sequence, naïve concatenation lets attention from tokens in A read tokens in B, an attention bleed that leaks information across document boundaries and silently degrades the model (it learns spurious cross-document correlations that never exist at inference, where each prompt is one document). The fix reuses the §05 attention-mask mechanism at the data level: reset position IDs at every document boundary and apply a block-diagonal causal mask so each document attends only to itself. The "Correct Mask Reset" mode shows the block-diagonal mask; the "Attention Bleed" mode shows the un-reset case where the mask is a single causal triangle spanning the whole packed sequence. The cost of the fix is a slightly more complex mask construction; the benefit is the $2\times$ throughput and the guarantee that packed training is information-equivalent to single-document training.

Verifier-guided model-in-the-loop synthesis

The defining innovation of Qwen3's data pipeline is automated verifier-guided synthesis: strong models generate missing capability distributions (such as multi-step Olympiad proofs and complex agentic refactors), but synthetic tokens are admitted to training shards only if validated by a deterministic oracle:

The verifier is the load-bearing column of Table 8, not the generator. Strong models generate candidate synthetic tokens cheaply and at arbitrary scale, but ungated synthetic data is model-flavoured noise: it amplifies the generator's own errors and, trained on at volume, causes the "model collapse" failure where the trained model drifts toward the generator's blind spots. The pipeline admits a synthetic token only if a deterministic oracle independently verifies it, and the oracle is deliberately not a language model: SymPy and Lean 4 check a math derivation by symbolic simplification and formal proof, not by judging whether the steps "look right"; Docker unit tests check code by execution pass/fail; agentic trajectories are checked by environment state-delta and exit-code assertion; OCR by bounding-box alignment and perceptual hashing. A deterministic oracle cannot be reward-hacked the way a model-judging-model verifier can, because there is no soft score to optimise against: the token either satisfies the oracle or it does not. The $14$ T synthetic can therefore be mixed with the $22$ T organic without degrading the base model: every synthetic token has passed a ground-truth check the organic data was never given.

The four capability axes in the table are exactly the distributions the organic $22$ T under-covers. Olympiad proofs, repository-level refactors, and multi-turn agentic error-recovery trajectories are rare in web crawl. A web corpus has millions of blog posts but vanishingly few verified step-by-step geometry lemmas. Synthesis closes those gaps: the generator produces the missing distribution, the verifier certifies which examples are correct, and only certified examples enter the shards. The $4.8$ T math, $5.6$ T code, $2.2$ T agentic, and $1.4$ T multimodal-OCR volumes sum to $14$ T, the synthetic term in the funnel equation above.

The two decontamination filters catch different failure modes, which is why both are required. The $13$-gram exact match catches verbatim leakage, a training document that reproduces a benchmark problem word-for-word, which the model would memorise into the §06 FFN slots and then "solve" by retrieval rather than reasoning at evaluation time. The cosine-similarity check catches paraphrase leakage, the same problem reworded, which a 13-gram match would miss but a semantic embedding would flag as near-identical. The $>0.85$ threshold is a precision-recall balance: too low and legitimately useful math content that merely resembles a benchmark problem gets purged (false positives starve the curriculum); too high and contaminated rewordings slip through (false negatives inflate the benchmark numbers the §13 results depend on). Purging is permanent and continuous. Every new shard is checked against the benchmark set before it reaches a training run, so contamination discovered late in a curriculum cannot have already been learned.

With the funnel distilled, the duplicates removed, the curriculum scheduled, the sequences packed, the synthetic gaps verified-closed, and the benchmarks decontaminated, the pipeline has produced $36$ T of training-ready tokens. The §10 pretraining stage now turns those tokens into the $27.58$ B trained weights: the loss schedule, the optimizer, and the compute that fills the architecture's memory slots with what the data contains.

Three stages, 36 trillion tokens, and 6.0 YottaFLOPs

Pretraining a 27.6B-parameter hybrid architecture is a staged three-phase curriculum spanning 36 trillion tokens, not unstructured brute-force gradient descent. Each stage changes the corpus composition, context length, learning rate schedule, and associative memory dynamics.

Three stages, three optimization regimes, not one long run

The curriculum is best read as three distinct optimization problems concatenated, not a single run with cosmetic phase labels. Each stage changes four knobs simultaneously: corpus mix (the §09 schedule), context length (which sub-system of the architecture is stressed: the FFN slots at 4k, the formal-reasoning circuits at 4k-STEM, the §03 DeltaNet recurrent state at 262k), learning-rate schedule (a fresh cosine decay per stage, restarted from an annealed peak), and associative-memory dynamics (the gradient noise profile that hits the 48 DeltaNet state matrices $S_t$). Stage 1 is broad-coverage language acquisition; Stage 2 is a STEM-and-code specialist that must not forget Stage 1; Stage 3 is a long-context extrapolation exercise that ladders the model from the 4k length it was trained on to the $262{,}144$ length it will be served at.

The transitions are deliberate shocks, and managing them is half the engineering. Stage 1→2 restarts the cosine LR from $1.2\times10^{-4}$ (down from Stage 1's $3.0\times10^{-4}$ peak), a warmup-and-decay that lets the model re-fit on the shifted STEM-heavy distribution without a catastrophic loss spike. Stage 2→3 ladders context $4\text{k} \to 32\text{k} \to 128\text{k} \to 262{,}144$ in steps, each rung engaging YaRN's RoPE base expansion ($\theta_{\text{base}} = 10^7$, the §05 extrapolation lever) at the new length, and drops LR to $3.0\times10^{-5}$ so the long-context fine-tuning does not perturb the dense knowledge already stored. The context ladder is itself a sub-curriculum: jumping straight to 262k would waste compute relearning what the 32k rung already teaches about position extrapolation.

The loss curve, read as three regimes

The four tracks in the trajectory widget below correspond to the four loss terms active during the curriculum, and each one maps to a stage. The general-language track dominates Stage 1: it descends smoothly as the model acquires syntax, multilingual grounding, and world knowledge, packing the §06 FFN slots with the long tail of facts. At the Stage 1→2 boundary the STEM/code track steepens (the cosine LR restart and the corpus pivot to formal reasoning accelerate reasoning-loss reduction) while the general track plateaus rather than rising, which is the empirical signature that the STEM specialist did not catastrophically forget Stage 1 (a plateau, not a spike, is the design goal). The long-context track only activates in Stage 3, descending as the 262k ladder teaches position extrapolation. The MTP auxiliary track (§08) runs throughout and descends independently, a regulariser that never spikes because it shares the backbone's hidden state.

The caption's "no late-stage divergence" is the payoff of the β₂ choice and the schedule discipline covered below, not luck. A hybrid model with 48 recurrent state matrices is numerically fragile to gradient noise, and a single stage transition that spikes the loss can leave $v_t$ inflated for hundreds of steps if the optimizer carries a long memory. The smooth descent across all four tracks at $36$ T tokens is the visible evidence that the fast-forgetting optimizer and the staged cosine schedule are doing their jobs.

The Chinchilla equation and the over-training decision

The total compute budget falls out of one multiplication. A forward pass costs $\approx 2ND$ matmul FLOPs ($N$ tokens each touching $\approx D$ parameters through the two projections of every layer), the backward pass roughly doubles it for the parameter and input gradients, and the MTP and optimiser overhead adds a small factor. The canonical approximation is $C \approx 6ND$. Plugging Qwen 3.8's numbers:

That $1300{:}1$ tokens-to-parameter ratio is the most consequential number in this section, and it is deliberately far above the $20{:}1$ that the Chinchilla scaling laws identify as compute-optimal. A model trained to Chinchilla-optimal on $27.6$ B parameters would stop at $\sim 0.55$ T tokens and gain the most capability per training FLOP, but it would also be the most capable model per training FLOP, not per inference FLOP. Qwen 3.8 instead over-trains a smaller dense model on far more tokens, trading extra one-time training compute for a permanently cheaper-to-serve model: at inference, a $27.6$ B model costs roughly a third the tokens/sec of an $80$ B Chinchilla-optimal equivalent while reaching comparable quality, which is the Llama-era "train a smaller model longer" insight. The $36$ T budget is the point where the over-training curve flattens enough that further tokens yield diminishing returns, and the calculator below lets you sweep it against $15$ T, $30$ T, and $50$ T to see where that knee sits.

The wall-clock version of the same arithmetic is what the calculator reports. Divide the $5.96\times10^{24}$ FLOPs by the cluster's sustained throughput (accelerator count $\times$ peak BF16 TFLOPs $\times$ model FLOPs utilisation (MFU), to get training seconds. For the default $4{,}096$ H100 SXM5 cluster at $42\%$ MFU, that is $5.96\times10^{24} / (4096 \times 989\times10^{12} \times 0.42) \approx 3.5\times10^{6}\text{ s} \approx 40\text{ days}$, and the widget recomputes this live for each hardware tier, GPU count, and MFU.

Optimizer dynamics & fast-forgetting gradient stability

The choice of $\beta_2 = 0.95$ (instead of the standard 0.999) is required in 64-layer hybrid models: In standard Transformers, $\beta_2 = 0.999$ carries a gradient second-moment memory half-life of $\approx 693$ steps. In a hybrid architecture where 48 layers carry recurrent associative state matrices $S_t$, a single noisy batch or localized loss spike can corrupt $v_t$ for nearly 700 steps, causing suppressed updates and numerical stalling. Setting $\beta_2 = 0.95$ drops the half-life to just $\approx 14$ steps, allowing the optimizer to rapidly flush stale gradient magnitudes and recover instantly from stage transition shocks.

The half-life figures are not heuristic: they fall out of the EMA structure. The second moment $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$ is an exponential moving average, so a single gradient shock decays as $\beta_2^{\,k}$ after $k$ steps. The number of steps for it to fall to half is the EMA half-life:

Why this matters more for a hybrid than for a pure Transformer is in the recurrent state. The AdamW update is $\eta \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$, so an inflated $v_t$ directly suppresses the effective learning rate: the denominator grows and the step shrinks. In a dense attention-only model a stale $v_t$ merely slows convergence; in the 48 DeltaNet layers, the state matrices $S_t$ accumulate across the sequence, so a gradient spike from one bad batch propagates into the state's gradient for many downstream positions, and a $\beta_2 = 0.999$ optimizer then suppresses exactly the corrective updates needed to flush that corruption, a self-reinforcing stall. The $0.95$ setting cuts the recovery to $14$ steps, which is fast enough that a stage-transition shock (the §10 schedule's intentional LR restarts) clears before the next checkpoint, keeping the loss curves in Fig. 6 smooth.

Distributed training & 3D parallelism topology

The 3D topology in Table 9 is not arbitrary: each axis maps onto a structural feature of the architecture and a physical feature of the cluster. Tensor parallelism splits the SwiGLU FFN's three projections and the attention Q/K/V heads inside one NVLink node, where the column/row-parallel all-gathers fit the $900$ GB/s intra-node bandwidth. Pipeline parallelism maps cleanly onto the §03 super-block structure: the $16 \times [3\Delta + 1\text{A}]$ periodic tiling gives $16$ natural pipeline units, so PP $= 4$ or $8$ places a super-block boundary at each stage cut, minimising the inter-stage activation traffic that crosses the slower InfiniBand links. Sequence parallelism only activates in Stage 3, where the $262$k context exceeds one GPU's memory: ring-attention shards the attention computation and the chunked WY-scan hands the DeltaNet recurrent state across GPUs, so the §04 training-parallelism trick now operates across an SP group rather than within one GPU's chunk grid. ZeRO data parallelism absorbs the rest of the GPUs by sharding the optimiser state, the $27.6$ B parameters' AdamW moments, so that the $\beta_2 = 0.95$ fast-forgetting state from the previous subsection is itself partitioned across the DP group. The four axes compose because they cut along independent dimensions (tensor, layer, sequence, data), and the super-block periodicity is what makes the layer cut clean.

Pretraining produces a base model that is knowledgeable but not yet useful, since it predicts the next token fluently but has not been taught to follow instructions, to reason step-by-step, or to refuse harmful requests. The §11 post-training stage takes that base model and aligns it, with GRPO reinforcement learning over verifier-scored reasoning traces.

From raw predictor to autonomous thinking agent

The post-training curriculum transforms a 27.6B foundation model into an autonomous, self-correcting reasoner across four synchronized phases: curated long chain-of-thought cold start, critic-free reasoning RL via Group Relative Policy Optimization (GRPO), thinking-mode fusion SFT, and general agentic alignment with explicit thinking-budget controls.

What post-training changes, and what it cannot

The §10 base model predicts the next token fluently but is not yet usable: it does not follow instructions, does not reason inside a delimited scratchpad, does not refuse harmful requests, and does not call tools. Post-training installs exactly these four behavioral surfaces across the four stages below. But there is a hard ceiling on what post-training can do, and it is set entirely by pretraining. Post-training shapes and reveals the knowledge already in the model; it cannot add knowledge the base model never acquired. The §06 FFN key–value slots were filled during the $36$ T-token curriculum of §§09–10, and RL only changes which slots fire and how the model expresses their contents; a verifier-scored GRPO step cannot teach a fact the FFN never stored.

The binding constraints on final quality are the §09 data funnel and the §10 compute budget, not the RL recipe. A perfectly aligned model on a weak base is a fluent liar; a poorly aligned model on a strong base is a capable but unruly predictor. The four stages below assume a strong base and convert it into a controllable, reasoning, tool-using agent, and each stage addresses a distinct gap that the previous one cannot.

Stage 1: Long-CoT cold start & reasoning scaffolds

A foundation model does not spontaneously produce disciplined, multi-thousand-token reasoning chains without prompt engineering. Stage 1 executes supervised fine-tuning (SFT) on a curated corpus of ~50,000 high-verification exemplars spanning Olympiad math, complex systems debugging, and algorithmic synthesis. This phase installs three critical behavioral priors:

• Delimited scratchpads: Encapsulates reasoning tokens inside <think> ... </think> blocks, allowing clean separation between raw deliberation and user-facing output.
• Subgoal decomposition: Enforces explicit problem restatement, edge-case identification, and branch planning before code generation.
• Hypothesis verification & backtracking: Installs linguistic patterns of self-critique ("Let me double check this edge case...", "This branch fails for negative inputs, resetting to...").

Cold-starting before RL is not optional, and the reason is the same one that made teacher forcing essential to the §08 MTP heads: RL explores, and it can only explore productively if its rollouts land in a region where rewards are reachable. A raw base model sampled at the temperatures Stage 2 needs would produce unstructured prose, and its $G$ rollouts would almost never hit a math verifier or pass a unit test, so the group-relative advantage signal would be pure noise, and the policy would drift toward whatever random phrasing happened to score once. The $50$k SFT traces install the $\langle\text{think}\rangle / \langle/\text{think}\rangle$ scaffold so that when Stage 2 samples $G$ rollouts, they are already structured reasoning attempts in the verifier's expected format, making the advantage signal meaningful. Cold start is the behavioural prior that makes RL's credit assignment tractable.

Stage 2: Group Relative Policy Optimization (GRPO)

Standard PPO requires training a separate Value Network (Critic) $V_\phi(s)$ of equivalent parameter volume to the policy network $\pi_\theta$, demanding an extra ~55 GB of VRAM during 27.6B training. GRPO (Group Relative Policy Optimization) completely eliminates the critic network. Instead, it generates a group of $G$ independent completions $\{y_1, y_2, \dots, y_G\}$ for each prompt $x$, evaluates their outcomes against deterministic verifiers, and normalizes their rewards across the group to construct the advantage baseline:

Eq.(15) and eq.(16) each do one job, and reading them as a control problem is the clearest path through the algebra. In classical PPO the advantage is $A = R - V_\phi(s)$, where $V_\phi$ is a learned value network the same size as the policy: the $+55.2$ GB critic GRPO deletes. Eq.(15) replaces that learned baseline with the empirical group mean: because all $G$ completions are sampled from the same policy on the same prompt $x$, their rewards $r_1 \dots r_G$ are samples of the same distribution, and $\text{mean}(r_1 \dots r_G)$ is a zero-shot, unbiased estimate of the prompt's intrinsic difficulty. Subtracting it cancels every confounding factor the completions share (problem hardness, vocabulary rarity, prompt idiosyncrasies), isolating the pure relative question "was $y_i$ better than its siblings?" The $\text{std}$ normalisation makes the advantage scale-invariant, so a hard prompt where every rollout fails (low rewards) and an easy prompt where every rollout succeeds (high rewards) produce the same advantage magnitude, preventing the optimiser from up-weighting easy prompts just because their absolute rewards are higher.

Eq.(16) is the PPO clipped surrogate with one structural addition. The ratio $r = \pi_\theta(y_i|x)/\pi_{\text{old}}(y_i|x)$ measures how much the policy has moved since the rollouts were collected; the $\min(r A_i, \text{clip}(r, 1-\epsilon, 1+\epsilon) A_i)$ caps the per-step update at $[1-\epsilon, 1+\epsilon]$ so a single lucky rollout cannot drag the policy into a destructive regime, the same trust-region principle as PPO. The $\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ term is the addition that does a different job: it anchors the policy to $\pi_{\text{ref}}$ (the pre-RL checkpoint) across the whole training run, preventing the cumulative drift that would otherwise push the model off the pretraining manifold and degrade its general English fluency (the "Distribution Drift" row of Table 10). The clip prevents per-step overshoot; the KL prevents cumulative drift. Each term is the fix for the failure mode its own omission would cause.

The $r_i \in [0,1]$ "from exact verifiers" clause is the same §09 principle. GRPO's advantage signal stays low-variance in a way reward-model RL cannot reach: the rewards come from deterministic oracles (a math answer check, a unit-test execution, an exit-code assertion), not a learned reward model. A learned reward model has blind spots the policy can exploit (the "Reward Hacking" row of Table 10); a unit test does not, because the policy cannot generate "tricky" code that passes a deterministic execution without actually being correct. The failure-mode mitigations in Table 10 are therefore verifier ensembles and sandboxing rather than reward-model retraining: the verifier side of the loop is deliberately kept non-neural.

Why group-relative baselines outperform monolithic critics

The group advantage in eq.(15) executes fundamental variance reduction: because all $G$ completions are sampled from the same policy state on the exact same input, they share all confounding factors (problem difficulty, vocabulary rarity, prompt idiosyncrasies). Subtracting the group mean $\mu_r$ cancels out the prompt baseline entirely, isolating the pure relative advantage: "Was completion $y_i$ superior to its contemporary sibling attempts?"

In Monte-Carlo terms, $\mu_r$ is a control variate highly correlated with $r_i$ (they share the prompt), so subtracting it reduces the variance of the gradient estimator without the bias a misspecified value network would introduce. This is the mathematical content of the "no critic drift" claim in the compare below. The cost is compute, not parameters: GRPO pays $G$ forward passes per prompt instead of one, trading the $+55.2$ GB critic VRAM for $G\times$ rollout throughput, which is why the rollout group parallelises across thousands of asynchronous inference workers rather than sitting in the trainer's memory.

Stage 3 & 4: thinking fusion & runtime effort control

To serve production environments requiring both deep multi-minute analytical reasoning and sub-100ms instant chat, Stage 3 merges both modes into a single checkpoint. Stage 4 applies a multi-reward stack (Helpfulness, Safety, Tool Calling, and Turn Persistence) parameterized by runtime effort knobs:

The thinking-budget explorer below makes a claim worth pinning down: that accuracy rises with reasoning effort up to a saturation knee, and that harder problems push the knee further out. The mechanism is retrieval. Each thinking token the model generates is a chance to surface a relevant §06 FFN memory slot or to retrieve a stored fact from the §04 DeltaNet recurrent state; harder problems need more such retrieval steps to assemble the solution, so more tokens convert into more accuracy until the relevant memory has been found. Past that point extra tokens add nothing (the saturation knee) and only cost compute. Trivial prompts reach the knee almost immediately (the answer is in the first retrieval), which is why the reasoning_effort knob exposes a runtime dial rather than a fixed schedule: the model does not pick it, the caller does, paying exactly for the retrieval depth a given prompt deserves. This is the inference-side analogue of §10's over-training decision (spend more compute at inference for better results without retraining), and preserve_thinking carries the scratchpad across agentic turns so that retrieval already done is not redone.

Post-training failure modes & stabilization matrix

Each row of Table 10 is a failure of one term in eq.(16), and the mitigation is the corresponding fix. Reward hacking is a failure of the reward source $r_i$ (the policy exploits a learned reward model's blind spots), fixed by replacing the model with deterministic verifier ensembles (the §09 principle). Entropy collapse is a failure of the eq.(15) denominator (when all $G$ rollouts converge to the same text, $\text{std}(r_1..r_G) \to 0$ and $A_i$ numerically explodes), fixed by the $\epsilon = 10^{-6}$ floor plus an entropy bonus $+\gamma H(\pi_\theta)$ that keeps the rollouts diverse. Distribution drift is a failure of the eq.(16) KL term (too small a $\beta$ lets the policy walk off the pretraining manifold and lose general fluency), fixed by adaptive $\beta$ scheduling that tightens the anchor as drift is detected. Length inflation is a failure of the reward's length-invariance (longer rollouts randomly stumble on reward triggers), fixed by subtracting $\lambda \log(\text{len})$ so raw length stops being rewarded.

Post-training yields an aligned model that reasons, refuses, and uses tools, and it still has to be served. A $27.6$ B model carrying a $262$k context window does not fit on a single consumer GPU at BF16, and the §12 optimizations are the engineering that closes that gap: quantization, paged KV management, chunked prefill, and the kernel fusions that make the architecture built in §§03–06 run at interactive latency.

The engineering that makes 262k context servable

Everything above establishes the architectural ceiling; this section details the deployment engineering that makes a 27.6B-parameter hybrid model servable on consumer GPUs: hardware roofline limits, quantization microscaling, paged KV management, chunked prefill scheduling, and kernel-level fusions.

Why serving is a memory problem, not a compute problem

At BF16 precision the 27.6B-parameter model weighs $55.2$ GB, more than any single consumer GPU holds, so the first reading of the serving problem is "the weights don't fit." That framing is correct but shallow: it treats VRAM as a bucket to be filled. The binding constraint at inference is subtler and lives on the memory bus, not in the silicon's FLOP budget. Single-stream autoregressive decode generates one token at a time, and to produce that one token the GPU must read every parameter exactly once from HBM across the bus. The work done per byte read, the arithmetic intensity, is what fixes whether the GPU is compute-bound or memory-bound, and for decode it is tiny.

Each parameter contributes roughly $2$ FLOP (a fused multiply-accumulate) to one token's forward pass, so the operational intensity of decode is the ratio of that work to the bytes that carry it:

The memory-bound and compute-bound regimes meet at the ridge point, $I^{\ast} = \text{peak FLOP/s} \,/\, \text{peak byte/s}$. For an RTX 5090 that is $1800$ TFLOP/s $\div$ $1792$ GB/s $\approx 1004$ FLOP/byte. Decode at $I_{\text{decode}} \approx 4$ FLOP/byte sits two and a half orders of magnitude left of the ridge, flat on the memory-bound ceiling. A $2\times$ faster Tensor Core does nothing for single-stream decode, because the compute units are already starved, waiting on the bus. Only three levers move the needle, and §12 is their engineering: (1) shrink the bytes read per token (quantization, the VRAM planner in Widget 1); (2) read the weights fewer times per emitted token (Multi-Token Prediction, the roofline simulator in Widget 2); (3) share one weight read across many concurrent requests (continuous batching and chunked prefill, the scheduler in Widget 3). All three work the same memory-bus bottleneck.

The VRAM budget, line by line

The planner above sums four terms. Each falls out of the architecture built in §§03–06, and reading them in order shows why quantization is a line-by-line decision rather than one global dial.

1. Model weights. The dominant fixed term. With $27.6$B parameters the footprint is exactly $\text{bytes}(W) \cdot N_{\text{params}}$, so the precision ladder is a clean $4\times$ descent:

The Q4 rung sits slightly above the naive $27.6 \times 0.5 = 13.8$ GB because the k-quant scheme carries a $6$-bit super-block scale per $256$ elements, the tax that keeps uniform $4$-bit from clipping outlier weights. This is the same dynamic the format cards below make explicit.

2. The GQA KV cache (the term that grows with context). Only the $16$ gated-attention layers accumulate a KV cache; the $48$ DeltaNet layers do not (they carry a fixed recurrent matrix, term 3). This is the §03 economics made concrete: bytes per attention layer scale with the context window $T$,

At $262$k and BF16 the cache ($17.18$ GB) exceeds the NVFP4 weights ($13.8$ GB). At long context the cache, not the weights, is the binding term, which is why the KV-cache dtype selector exists independently of the weights selector. Cutting the cache from BF16 to FP8 to NVFP4 halves, then quarters, this term directly: $17.18 \to 8.59 \to 4.29$ GB. This is the §03 KV-cache derivation driving the §12 quantization decision: the attention layers' cache is the one term that grows without bound as $T \to 262$k and beyond, so it is the one term where precision choice has a context-dependent payoff.

3. The DeltaNet recurrent state (fixed and tiny). The $48$ DeltaNet layers each hold a recurrent matrix $S$ of $48$ V-heads $\times$ $128 \times 128$, kept in BF16 regardless of the weights format (the never-quantize callout below). Its footprint is

This single number is the architectural payoff of §04. Forty-eight of the sixty-four layers, three quarters of the depth, consume $75.5$ MB of state no matter how long the context, against the attention layers' $17$ GB at $262$k. The hybrid's servability on a $24$ GB GPU comes from this $226\times$ ratio between the linear-attention state and the full-attention cache, built into the layer recipe in §03, not from a quantization trick.

4. Runtime overhead. The CUDA context plus activation buffers, modelled as $1.5 + 0.1\,B$ GB, a near-constant tax that scales only mildly with concurrency $B$. It sets the floor below which no precision choice helps.

Stacking the default configuration (NVFP4 weights, FP8 KV, $T\!=\!131$k, $B\!=\!1$, $24$ GB target) gives the planner's headline verdict, derived rather than asserted:

Why NVFP4 rather than uniform INT4 is the rung the production config stops at: microscaling attaches an FP8 scale factor to every block of $16$ elements, so a weight outlier is carried by its own local scale rather than clipping the whole tensor. Uniform $4$-bit has one global dynamic range and loses the outlier, and the §06 FFN key–value memory is precisely where outlier weights encode the rare, high-magnitude records the model retrieves at inference. The two-level block format runs natively on the 5th-generation Tensor Cores (Blackwell/Ada), so the $4\times$ footprint cut costs no kernel dispatch overhead. The format cards below unpack the bit layouts that make this work.

The roofline, read as two regimes

The plot above is a single diagram with two ceilings and one ridge; decoding which side a workload falls on tells you exactly which upgrade buys throughput. On the left, the memory-bound ceiling is the flat line at $\text{peak byte/s}$. Performance rises linearly with arithmetic intensity until it hits the compute-bound ceiling, the flat line at $\text{peak FLOP/s}$, on the right. The ridge point $I^{\ast}$ where they meet is the operational intensity at which the GPU saturates both buses at once.

Decode lives on the memory-bound ceiling. As derived above, single-stream decode at NVFP4 runs at $I_{\text{decode}} \approx 4$ FLOP/byte, two and a half orders of magnitude left of the $5090$'s $I^{\ast} \approx 1004$. On the memory-bound ceiling, throughput is the bandwidth divided by the footprint, the simulator's baseline rate:

On this ceiling the lever is unambiguous: halve the footprint (NVFP4 vs FP8) and the rate doubles; double the bandwidth (H100's $3350$ GB/s) and the rate doubles; double the Tensor Core FLOP/s and the rate does not move. The §08 LM-head bandwidth point bites here for exactly this reason. The $248{,}320$-vocabulary output projection is a $1.27$B-parameter matrix ($248320 \times 5120$), and at single-batch decode it is read in full every token to produce one distribution. That one matmul is pure HBM traffic, the largest single weight read in the forward pass, paid once per token.

Prefill lives on the compute-bound ceiling. During prefill the same weights are reused across all $T$ prompt tokens at once, so the arithmetic intensity scales as $\sim T \cdot I_{\text{decode}}$ and moves right of the ridge. Prefill is limited by peak FLOP/s, which is why its optimization is FlashAttention-3 and fused kernels (Table 11's right column), not quantization. The two columns of Table 11 (prefill complexity vs decode complexity) are the two regimes of this one roofline: prefill wants TFLOPs, decode wants GB/s.

MTP is a roofline move, not a compute speedup. Multi-Token Prediction does not make any single matmul faster; it changes how many tokens one weight read buys. The draft–verify–accept cycle from §08 reads the $1.27$B LM head (and the rest of the weights) once and produces $K+1$ candidate tokens, so the effective per-token cost of that bandwidth-bound read is divided by the expected tokens accepted per pass:

The simulator multiplies the baseline by this speedup: $130 \times 2.12 \approx 276$ tok/s. Read it as "MTP pushed the LM-head matmul from $I \approx 4$ toward $I \approx 8.5$ FLOP/byte on the roofline," reusing each weight read across the accepted tokens, not as "MTP made the GPU $2.12\times$ faster," since the silicon is unchanged. It is a bandwidth optimization that only works because decode is memory-bound; on the compute-bound ceiling it would buy nothing. MTP also helps decode and only decode: it amortizes the per-token weight read. It does nothing for the moment a long prompt first arrives, which is prefill, and prefill is scheduled, not speculated.

Chunked prefill: why a long prompt can't arrive all at once

MTP accelerates decode; it does nothing for the moment a $131$k-token prompt is submitted. That moment is prefill, the forward pass that ingests the whole prompt, computes attention across it, and materializes its KV cache before the first token can be generated. Prefill has the opposite profile from decode: it is compute-bound (each weight reused $T$ times) and it is also the pass that must lay down the entire KV cache at once. Naively, a $131$k prompt arriving into an idle GPU does two damaging things: it spikes the $O(T^2)$ attention compute of the $16$ full-attention layers across the whole prompt in one burst, and it blocks the GPU for that whole burst, so a short conversational decode request queued behind it waits $T_{\text{prefill}}$ milliseconds before its first token. That head-of-line blocking is the latency spike the scheduler exists to remove.

Chunked prefill splits the prompt into $8$k–$16$k token chunks and runs one partial prefill per chunk (each chunk extends the KV cache by its own length and attends to the already-cached prefix) interleaved with single-token decode passes for live requests. Two things change. First, the KV cache grows incrementally ($8$k at a time) rather than materializing in one shot, so the peak working set is bounded by the chunk size, not the prompt length. Second, the scheduler gives decode passes priority between chunks, so a short query is never made to wait behind a long prefill. The total prefill FLOPs are not reduced. The $O(T^2)$ attention over the prompt is invariant to how it is sliced, and FlashAttention-3 already keeps prefill memory at $O(T)$, so chunking is fairly characterized as a scheduling and working-set bounding optimization, not a FLOP reduction. Its win is latency fairness: the naive mode in the widget above shows the pipeline bubble where prefill monopolises the GPU; the continuous mode shows the interleaved timeline where decode slips through the gaps.

The third leg of the scheduler is paged KV allocation. The cache is laid out in fixed $16$-token pages rather than one contiguous buffer per sequence, so allocation grows in small bounded increments and fragmentation stays below one page. Because identical prefixes produce identical KV blocks, a radix tree over the token stream lets a shared system prompt's KV be stored once and referenced by every request that uses it. The §03 KV-cache economics applied across requests rather than across layers. Where GQA in §05 deduplicated KV heads within a layer, paged prefix-sharing deduplicates KV blocks across a batch. Both are the same principle applied at two different scales: the cache holds reusable state, not private state. Together, chunked prefill and paged KV are what let a $27.6$B model with a $262$k window serve a mix of long documents and short chats on one GPU without the long documents starving the chats.

Quantization microscaling & numerical precision

The ladder above conceals an essential numerical insight: why does FP8 and NVFP4 cost negligible benchmark accuracy, whereas naive uniform 4-bit integer quantization degrades reasoning? The difference lies in microscaling block structures and floating-point exponent budgets:

Long-context computational scaling comparison

None of this changes what the model knows. The knowledge substrate is the $1.11$M FFN key–value records laid down in §06 and the recurrent state written in §04; the §10 compute budget fixed their content. The §12 engineering (quantization, roofline-aware scheduling, MTP amortization, chunked prefill) only changes how cheaply and how fast that fixed substrate can be read at inference. The one question it raises and cannot itself answer is whether the numerical compression that makes the model fit a $24$ GB GPU also preserves the accuracy the pretraining budget paid for. That is the question §13's benchmarks are run to settle.

Empirical capability & benchmarks

Benchmarks are not the point of this walkthrough (the architecture is), but the empirical record is the external check that the mechanisms of §§03–11 actually compound into capability. Read architecturally, each benchmark domain is a probe of a different substrate built earlier in the document: knowledge probes retrieve from the §06 FFN key–value memory, long-horizon agents lean on the §04 recurrent state held across thousands of steps, multimodal tasks route through the §07 SigLIP tower and §05 MRoPE, and reasoning tasks spend §11 thinking tokens to surface what pretraining stored. The four headline deltas below are where the hybrid Gated DeltaNet backbone, the native vision front-end, and the GRPO reasoning RL show up as double-digit jumps, most sharply in long-horizon software engineering and autonomous computer use.

The detailed vendor scorecards that filled this section (radar, parity scatter, margin waterfall, Pareto frontier, and the master catalog) have been removed: for an architectural walkthrough they were comparison shopping, not instruction. What remains is the one view that earns its place, the mapping from each capability domain to the architectural component that delivers it.

The whole hybrid architecture in standalone PyTorch

Every equation from §02–08 assembled into clean, standalone, fully runnable PyTorch code. Features zero opaque dependencies, explicit tensor shape annotations throughout, and an interactive tensor tracer that computes memory buffers and FLOPs for any batch size or sequence length.

A verifiability artifact, not a serving kernel

The code below is deliberately small (roughly 340 lines hold the whole 27.6-billion-parameter architecture) because it is a read-through of the model, not a deployment. Every line maps to an equation from §§02–08; nothing is hidden inside a fused kernel, a CUDA graph, or a library call whose internals the reader cannot see. That choice buys total verifiability at a well-defined price: the code is correct about what the model computes and how the tensors flow, but it is intentionally naïve about how fast that compute runs. The DeltaNet layer steps its recurrent state in a Python for t in range(T) loop; the attention layer calls F.scaled_dot_product_attention without tiling its own KV; the rotary path falls back to one-dimensional positions. Each of these is a production shortcut removed, and each omission points back to the section that owns the real kernel.

Read the two widgets as a pair. The tracer above walks one subsystem's tensors shape-by-shape and reports the memory and FLOP bill the code actually incurs; the studio below gives the source itself, file by file, with each module cross-linked to its derivation. The contract table that closes the section lists the invariants the code is bound by, and the prose between the studio and the table walks the source so that each one is derived from the config.

Reading the config: one object, every constant sourced

Qwen38Config is the single source of truth for the entire codebase. Every dimension in the four modules below is read off it, and every number in the section's widgets and tables is a few multiplications of its fields. The backbone width d_model = 5120 is the §01 residual bus made concrete; the hybrid ratio "3:1" and n_layers = 64 are the §03 super-block tiling; the DeltaNet head geometry (16, 48, 128) is the §04 address/content split; the attention (24, 4, 256) is the §05 GQA compression; d_ff = 17408 is the §06 3.4× expansion; mtp_k = 2 is the §08 draft depth; vocab_size = 248320 is the §02 tensor-core-aligned vocabulary. Nothing in the forward pass is a free parameter that the config does not already name.

The per-layer parameter budget falls directly out of these fields. A DeltaNet mixer carries one fused QKV projection, a small write-gate projection, and an output projection; an attention mixer carries Q, K, V, gate, and output projections; the SwiGLU FFN carries three matrices. Working the three costs out of the config gives the labor division in parameter units:

Two things are already visible before any tensor moves. First, the mixer is the cheap part: even the wider attention layer costs 105 M parameters, less than half the FFN that sits beside it on every one of the 64 layers. Second, the DeltaNet layer is cheaper than the attention layer (84 M vs 105 M) despite doing the harder job of holding a recurrent state, so the 3:1 ratio is a parameter win as well as a memory win. Stacking one super-block (three DeltaNet, one attention, four FFNs) gives $3(84.1+267.4) + (104.9+267.4) \approx 1427\text{M}$, and sixteen super-blocks put $\approx 22.8\text{B}$ parameters in the 64 backbone layers. The embedding and untied LM head add $\approx 2.54\text{B}$ and the MTP module $\approx 1.32\text{B}$, taking the language model to $\approx 26.7\text{B}$; the §07 vision tower, absent from this code, closes the gap to the 27.6 B figure on the model card.

The two mixers, one residual bus

HybridSuperBlock.forward is six lines that encode the whole §03 thesis. Each of the three DeltaNet sub-layers and the single attention sub-layer follows the same recipe: normalize the stream for the mixer (norm1), run the mixer, add its output back to the raw stream, then normalize for the FFN (norm2), run the FFN, and add that back. The mixer routes information across positions; the FFN transforms each position independently; both read from and write to the same 5120-wide residual channel through Pre-RMSNorm. This is the §06 bus-arithmetic contract in code: every reader sees a unit-scale view of the stream, and the unnormalized bus accumulates all of them additively with no contention.

The only asymmetry between the four sub-layers is which mixer they instantiate, and that single switch, repeated sixteen times, is the entire 3:1 hybrid pattern. There is no routing, no learned mixer selection, no per-layer flag; the motif is hard-wired by hybrid_pattern = "3:1" and tiled by range(3) followed by one attention layer. What looks like a loop is the §03 compression–anchor cycle in literal form: three layers of constant-space recurrence, then one layer of exact global attention to anchor what recurrence compressed.

The delta-rule scan, read line by line

GatedDeltaNet.forward is the densest module and the one where the shapes matter most. The fused in_proj emits a single $[B, T, 10240]$ tensor that torch.split carves into Q, K (each $[B, T, 16, 128]$) and V ($[B, T, 48, 128]$). The 1:3 address-to-content asymmetry from §04 is then restored by repeat_interleave(3, dim=2), which triples each of the 16 normalized key/query addresses to face the 48 value heads, so three content slots share one address without tripling the address space. F.normalize(p=2, dim=-1) applies the §04 stability invariant: every key and query is unit-norm before it touches the state, so no outer product can expand the state's spectrum.

The two gates are one-liners that carry the §04 mechanism. beta = 2.0 * sigmoid(...) bounds the write strength in $(0, 2]$, where the cap at 2 is the contraction bound that keeps $I - \beta k k^{\top}$ non-expanding when $k$ is unit-norm. alpha = exp(-softplus(a_log).exp()) bounds the decay in $(0, 1)$, strictly dissipative, with the decay rate itself learned per value head so forgetting is content-aware rather than on a fixed schedule. The recurrent loop then reads as the §04 error-correcting write in three tensor ops per step: predict the stored value $\text{pred}_v = S_{t-1} k_t$, form the error $\text{err} = \text{pred}_v - v_t$, and overwrite $S_t = \alpha S_{t-1} - \beta\,(\text{err} \otimes k_t)$, a rank-one projection that removes the old slot and stamps the new one in the same operation.

That loop is correct and it is the whole point of the module, but it is also the place where the reference parts company with a serving kernel. A Python-level scan over $T$ steps gives the right recurrence but cannot use Tensor Cores, since each step is a $128\times128$ matmul launched from the interpreter. Production training replaces this loop with the chunked WY factorization of §04, which collapses $C$ rank-one updates into one block-triangular matmul and carries inter-chunk state in FP32. The reference keeps the loop because the loop is equation (3)–(4); the factorization is a §04 detail that would hide the very mechanism it exists to expose.

MRoPE and the growing cache

GatedAttention.forward mirrors the DeltaNet structure up to the mixer. Q, K, V, and a gate are projected; K and V share only four heads ($[B, T, 4, 256]$) while Q has twenty-four, so repeat_interleave(6, dim=2) expands them into the §05 6:1 GQA compression, six query heads served from one cached key–value pair. The rotary path is the one place the reference simplifies in the open: _apply_mrope builds a one-dimensional frequency table and rotates the first 64 of 256 head channels, leaving the remaining 192 untouched, which is the partial-RoPE contract from §05. The full MRoPE [11, 11, 10] split that assigns independent time/height/width axes to those 64 rotary channels is a frequency-table construction, not a different rotation, so the 1D fallback shows the rotation math and the 3D path is left as the §05 table that reweights which channels belong to which axis.

The cache is the second half of the §05 story and the one the contract table weighs against the DeltaNet state. On every decode step torch.cat([kv_cache[0], k], dim=1) grows the K and V tensors by one position, so the per-layer buffer is $[B, 4, T, 256]$ and its cost is linear in $T$, unlike the DeltaNet state, which is a fixed $[B, 48, 128, 128]$ regardless of how long the sequence runs. The output gate attn_out * gate is the last line of the module and the §05 quality-compensation mechanism: a data-dependent sigmoid suppresses the noisy matches that four shared KV heads inevitably produce before the output projection ever sees them.

MTP: one shared head for K drafts

MultiTokenPredictionHead is the smallest module and the one with the sharpest parameter accounting. Each of the $K=2$ draft blocks is a single RMSNorm plus a $5120\times5120$ linear, and all $K$ drafts share one $5120\times248320$ vocabulary projection rather than each owning a private copy. The cost is therefore $K\cdot D^2 + D\cdot V \approx 2\cdot 26.2\text{M} + 1.27\text{B} \approx 1.32\text{B}$ parameters, dominated by the shared head, which is why sharing it across drafts (and reusing the backbone's own LM head in production) is the §08 efficiency move. The forward pass returns a list of $K$ logit tensors, one per draft depth, all read from the same final hidden state; the §08 draft–verify–accept loop that consumes them is inference-harness code, not model code, so it correctly lives outside this module.

The invariants, read off the code

The contract matrix above is not an external claim about the architecture. It is a set of arithmetic facts the code already encodes. Worked out of the config, they line up with every earlier section without a single new assumption. The DeltaNet state is a fixed $n_v \cdot d_h^2$ matrix per layer, so its size is invariant in $T$ and the only question is the precision and how many layers one counts:

The two state figures that appear elsewhere in the document, Table 13's "3.15 MB per layer (FP32)" and the §12 budget's "75.5 MB total (BF16)", are therefore the same number in two framings: per-layer versus all-layer, FP32 versus BF16. There is no discrepancy to reconcile; there is one invariant quantity reported at two scales.

The attention KV cache is the same calculation with a $T$ in it. Each attention layer stores two tensors, K and V, over four heads of width 256, so its footprint grows linearly with the context it has seen:

Set these two against each other and the whole hybrid thesis falls out of the config: at the full 262k context the sixteen attention layers cost 17.18 GB and the forty-eight DeltaNet layers cost 75.5 MB, a factor of $\approx 227\times$, over two orders of magnitude. That single ratio is why the architecture fits on a 24 GB consumer GPU (§12) and why the contract table lists DeltaNet decode as $O(1)$ against attention's $O(T)$:

The prefill row tells the same story at the other end of the sequence. DeltaNet threads the recurrence for $T d_h^2$ work, linear in $T$, with the $d_h^2 = 16{,}384$ constant set by the fixed head matrix; attention builds the full $T \times T$ score matrix for $T^2 d_h$ work, quadratic in $T$. The two cross over at $T = d_h = 256$: below a few hundred tokens they are comparable, and above it the quadratic pulls away hard, which is exactly why the contract table lists DeltaNet prefill as $O(T)$ against attention's $O(T^2)$. The table's two complexity columns are this crossover, read off the config.

What the code deliberately omits

The reference is a read-through, so each production feature it leaves out is a section it points back to. The DeltaNet scan is a Python loop, not the chunked WY factorization that lets the recurrence use Tensor Cores (§04). The attention path calls F.scaled_dot_product_attention rather than tiling its own KV, so the $O(T^2)$ memory the contract table records is the mathematical truth, while the $O(T)$ memory FlashAttention actually achieves is a §12 kernel fact. The MRoPE path builds a 1D frequency table and leaves the 3D [11, 11, 10] axis assignment as the §05 position construction. The KV cache is a plain tensor, not the 16-token paged blocks with radix-tree prefix dedup that §12 uses to share system-prompt state across requests. The MTP module emits draft logits but the draft–verify–accept loop that consumes them is §08 harness code. And the §07 SigLIP vision tower is absent entirely. This is the language backbone, the part of the model that runs after pixels have already become bus-native tokens.

None of these omissions weakens the code as a specification of the model; they strengthen it as a teaching artifact, because every removed shortcut is a place where production engineering, not architecture, does the work, and each is already documented in its own section. The result is a description of what Qwen 3.8-27B computes that runs on any machine with PyTorch installed.

Conclusions