FIG · A0 / PARAM 434M ACTIVE / TOTAL 1.13B / GDN 24+MLA 8 (3:1) / MoE 16+1 TOP-2 / DEPTH 32 / MTP 2

HyMo-434M — documentation portal

A 434M-active / 1.13B-stored hybrid language model: Gated Delta Networks (linear attention) interleaved 3:1 with Multi-Head Latent Attention (full attention), routed through a 16+1 Asymmetric Mixture-of-Experts. Pure PyTorch — one hand-written Triton GDN recurrence kernel, a NorMuon + AdamW dual optimizer, and a 30B-token training target.

HYBRID RATIO

75% SUB-QUADRATIC

3:1 stack ratio · 24 GDN linear layers + 8 MLA full-attention layers

ASYMMETRIC FFN

16+1 TOP-2 MoE

Sparse MoE (2304d) on 8 MLA blocks; 24 GDN blocks are recurrence-only (no FFN)

DUAL OPTIMIZER

NorMuon + AdamW

NorMuon for 2D matrix weights; Cautious AdamW for 1D, Embeddings & MoE

MTP PREDICTION

2 HEADS [0.3, 0.1]

Multi-Token Prediction auxiliary heads accelerate representation learning

FIG. A0 HYBRID RECURRENCE FIELD · GDN · MLA · MoE 32 LAYERS · 896 DIMS · 16 HEADS

SIMULATION MODE GDN · 1D SELECTIVE SCAN (O(N))

STATE FOOTPRINT O(1) STATE (O(N) COMPUTE)

BLOCK ALLOCATION 24 LAYERS (DENSE SwiGLU)

KERNEL DISPATCH FUSED TRITON (CHUNK 64)

PROBE · GDN HEAD #04 State Decay α = 0.942 · Chunk #04 Recurrence Update Δh = (1 - β q k⊤) h + β v k⊤

h_t = (1 - β_t q_t k_t^⊤) h_t-1 + β_t v_t k_t^⊤ · α_t = exp(g_t A)

24 GDN Linear Layers 8 MLA Full Attention Layers 16+1 Asymmetric MoE / MTP CLICK: SHOCKWAVE · HOVER: PROBE HUD

01 · params: ~750M
02 · layers: 32 24+8 (3:1)
03 · attention: MLA 4 KV grp
04 · recurrence: GDN chunk 64
05 · ffn: 16+1 MoE / SwiGLU
06 · vocab: 64,256 BPE-64k
07 · ctx & rope: 4096 25% decoupled
08 · optimizer: Dual NorMuon+AdamW

§ HYMO CORE MECHANISMS · INTERACTIVE BENCHMARK LABS

01 · GDN RECURRENCE Fused 1D Selective Scan

Gated Delta Net processes linear recurrence with sub-quadratic O(N) complexity. Learned decay α_t = exp(g_t A) and write/read state decomposition.

Recurrence Chunk Size: 64 tokens / chunk

DELTA RULE h_t = (1 - β_t q_t k_t⊤) h_{t-1} + β_t v_t k_t⊤

TRITON KERNEL Fused 1D selective scan with chunked recurrence (Q=64)

Sub-Quadratic Ratio: 24 of 32 Layers (75% Stack)

                                            Numerical Parity:
                                            < 1e-5 max |Δ| vs Eager
                                        

02 · ASYMMETRIC MoE 16+1 Router on MLA Layers

DeepSeekMoE with 16 routed experts (2304d) + 1 shared expert (2304d). Top-2 routed per token with aux-loss-free dynamic bias leveling Δb.

Active Experts: #02, #07 + shared (2304d)

                                            FFN Placement:
                                            MLA Only (GDN blocks are recurrence-only, no FFN)
                                        

03 · MLA COMPRESSION 4 KV Groups Footprint Analyzer

Multi-Head Latent Attention compresses KV cache to 4 groups (128 latent + 32 rope dims). Reduces KV cache footprint by 16× per layer vs standard MHA.

Context Length: 32,768 tokens

Standard MHA (16 heads × 128d × 32L): 8.00 GB

HyMo Hybrid (24 GDN + 8 MLA): 0.13 GB

                                            KV Cache Memory Saved:
                                            7.87 GB (16× cut per layer)
                                        

04 · MTP SPECULATION Depth-2 Speculative Tree

Multi-token prediction. Main 32-layer backbone predicts token t while depth-1 and depth-2 modules draft candidate tokens t+1 and t+2 with shared embeddings.

MAIN HEAD Token t → "architectures" (p = 0.99)

MTP DRAFT 1 Token t+1 → "converge" (p = 0.88 ≥ 0.80)

MTP DRAFT 2 Token t+2 → "faster" (p = 0.82 ≥ 0.75)

Speculative Verification: ACCEPTED (3 Tokens / Step)

                                            Throughput Acceleration:
                                            2.00× effective speedup
                                        

FIG. A1 FULL TRAINING STEP PIPELINE 32 LAYERS · FORWARD · AUTOGRAD · DUAL OPTIMIZER

CURRENT PHASE AUTO CYCLE (FWD → BWD → DUAL OPT)

MEMORY CONTRACT FSDP-2 BF16 · 75% SUB-QUADRATIC · FP32 MASTER

• FORWARD · Tokens x_t → Embed → 24× GDN (Linear) ⇄ 8× MLA (MoE 16+1) → MTP Heads → CE Loss → NorMuon/AdamW

Forward Activations Backward Gradients (∂ℓ/∂θ) NorMuon + AdamW Step CLICK STAGE: INSPECT · HOVER: TENSOR SHAPES

§ core

Core Architecture

5 entries

README

Project Overview

434M-active / 1.13B-stored hybrid language model: GDN × MLA with Asymmetric MoE.

AGENTS

System Architecture

Codebase contracts, Triton rules, typing invariants & architectural limits.

SKILLS

Skills Map

Specialized agent tools, scripts, and domain competencies.

DOCS

Documentation Index

A map of the concepts, guides, and API references in this portal.

TRAIN

Training Pipeline

FSDP-2 sharding, NorMuon + AdamW dual optimizer, WSD schedule, DCP checkpointing.

§ concepts

Architecture & Concepts

5 entries

Model Architecture

Line-by-line code walkthrough of model, GDN, MLA, MoE, MTP, and partial RoPE.

GDN & MLA Mechanisms

Linear attention delta rule, latent KV compression, 3:1 hybrid stack thesis, and MoE-on-attention-only FFN split.

Optimization Quartet

NorMuon for 2D matrices, AdamW for 1D/embeddings/MoE, WSD decay, and FSDP-2 sharding.

Operations & Triton

Custom Triton GDN kernel: fused 1D selective scan with chunked recurrence (chunk=64).

v1.0 Design Spec

The complete architectural specification, scaling decisions, and validation criteria.

§ guides

Guides & Playbooks

1 entries

Quickstart

From zero to a running forward pass — install, smoke tests, gates, and training loop.

§ refs

API References

2 entries

Typed Config Reference

Complete specification for ModelConfig, OptimizerConfig, SchedulerConfig, and validation.

Public API Reference

HyMo model interface, Trainer, optimizers, schedulers, and factory entry points.