GPT-OSS-Lite Model Architecture

GPT-OSS-Lite Model Architecture An architecture diagram generated by Archify. Input IDs · (B, T) int64 · Architecture component Input IDs (B, T) int64 Token Embedding · 128000 rows · width 768 · Architecture component · weight tied Token Embedding 128000 rows · width 768 weight tied Norm → Attention → Add · 8 Q / 4 KV · head dim 96 · GPTOSSBlock ×12 · Attention THEN MoE in EACH block · even SWA 128 / odd full Norm → Attention → Add 8 Q / 4 KV · head dim 96 even SWA 128 / odd full Norm → MoE → Add · top-2 of 8 + 1 shared · GPTOSSBlock ×12 · Attention THEN MoE in EACH block · SwiGLU · ffn 1536 Norm → MoE → Add top-2 of 8 + 1 shared SwiGLU · ffn 1536 Norm + Tied Head · 768 → 128000 logits · Architecture component · native dtype Norm + Tied Head 768 → 128000 logits native dtype Model Outputs · logits (B,T,V) + mean aux · Architecture component Model Outputs logits (B,T,V) + mean aux ModelConfig · A100 YAML + defaults · Architecture component ModelConfig A100 YAML + defaults YaRN RoPE · global: 24/48 pairs off · Architecture component · local: all 48 pairs YaRN RoPE global: 24/48 pairs off local: all 48 pairs MixedKVCache · rotated K · unchanged V · Architecture component · decode-only state MixedKVCache rotated K · unchanged V decode-only state token lookup after block 12 next-token logits validated dimensions rotate Q/K decode append GPTOSSBlock ×12 · Attention THEN MoE in EACH block Legend Backend Database External

Repeated composite

  • • Even blocks 0,2,…10: SWA. Odd blocks 1,3,…11: full.
  • • Each sublayer has its own pre-norm and residual addition.

Logical cache, not physical VRAM

  • • At 131072 tokens: 12T / (6T + 6×128) = 1.998×.
  • • Excludes global spare capacity, ring duplication and temporary tensors.

Source / configured baseline

  • • Revision 3c9675deaf16
  • • configs/pretrain_a100_502m.yaml
  • • ModelConfig defaults · no CLI overrides