Overview
Source baseline: 3c9675d, configs/pretrain_a100_502m.yaml plus ModelConfig defaults. See evidence below for exact hashes and independent validation, browser and perceptual status.
Model Architecture 🏛️
12-layer transformer backbone showing alternating sliding-window and full self-attention blocks, one Attention+MoE composite repeated 12 times. Global YaRN disables 24 of 48 frequency pairs, not 25%. Decode stores rotated K and unchanged V.
docs/gpt-oss-lite-model-architecture.html
Open Diagram ↗
Data Pipeline ⚡
Declared 8B-token recipe, shared packing and mmap-backed windows. EOS 128009 exceeds the configured 128000 embedding rows. This integration gap is unresolved, not a working producer-to-model handoff.
docs/gpt-oss-lite-data-pipeline.html
Open Diagram ↗
Training Workflow 🔄
Pure PyTorch pretraining loop featuring mixed-precision BF16 forward pass, auxiliary load-balancing loss, CE before the finite-loss gate, four-micro-batch accumulation and optimizer-clock checkpoints. Final save adds RNG separately. Neither periodic nor final saves capture sampler position. Five nonfinite losses restore the latest checkpoint or abort when none exists.
docs/gpt-oss-lite-training-workflow.html
Open Diagram ↗
Optimization Stack 🚀
Separate host, activation, inference-cache, compute and optimizer-state budgets. Compilation and SDPA dispatch are conditional. Periodic checkpoints omit RNG and sampler position.
docs/gpt-oss-lite-optimization-stack.html
Open Diagram ↗
KV-Cache Memory & Alternating Attention Calculator
Logical BF16 K/V entries only, not measured VRAM. Uses 12 layers, 4 KV heads and head_dim=96. Excludes global spare capacity, ring duplication and temporary tensors. Learned sink logits do not reserve four cached tokens.
Top-2 of 8 MoE Expert Routing & Load Balancer
Illustrative logits, not learned expert specializations. Bars show full softmax probabilities, not renormalized routing weights. Temperature is a teaching control, not a production router setting.
Architecture Specifications
| Pillar | Specification | Source Reference |
|---|---|---|
| Backbone | 12-layer Transformer, d_model=768, n_heads=8, n_kv_heads=4, d_head=96 | models/transformer.py:GPTOSS |
| Attention Pattern | Alternating: Even layers 128-token SWA + learned per-head sink logit; Odd layers Full Attention | models/attention.py:GPTOSSAttention |
| Context Scaling | YaRN RoPE interpolation up to 131,072 (128K) tokens | models/yarn.py:YaRNRoPE |
| Feed-Forward / MoE | 8 Routed Experts + 2 Active per token, 1 Shared Expert, SwiGLU ffn_dim=1536 | models/moe.py:MoELayer |
| Training Target | 8.0B Chinchilla-optimal tokens, single A100-80GB (16–20h unverified target) | training/pretrain.py:main |
Delivery Receipts & Artifact Hashes
All delivered HTML artifacts are compiled and verified with SHA-256 bindings. Fresh diagram browser checks pass separately from visual review, which is unavailable. Measured secondary text minima are 7–9 CSS px, below the proposed 11px target. Guide mobile checks and viewer interactions/exports remain unverified. For exact source/config pins and limits see RECEIPTS.md.
| Artifact | Type | SHA-256 Hash | Byte Size |
|---|---|---|---|
gpt-oss-lite-model-architecture.html | architecture | 632f1cd9b84271d8d4c69da9fa623d31fd557daccb53f1ffed2b08fd1713fc10 | 715,968 B |
gpt-oss-lite-data-pipeline.html | dataflow | 79ee1da4b1e64357d264cf115faf803ef555312ef60ca40ac6e2499177fdfb35 | 714,822 B |
gpt-oss-lite-training-workflow.html | workflow | 396a615b33a8c614edc7590404b963b8fb68822cf875b713a7897e96eedd1a5f | 719,036 B |
gpt-oss-lite-optimization-stack.html | architecture | 28fe1e481ffd50f6e2ae17243b9d18621961c5f08f582d2268caf47bb45c6d06 | 710,291 B |