GPT-OSS-Lite Data Pipeline

GPT-OSS-Lite Data Pipeline A data-flow diagram generated by Archify. 01 / Recipe 02 / Acquire 03 / Curate 04 / Encode + Pack 05 / Train Declared 8B Recipe · 7 weights, not output quotas · 01 / Recipe · mixture.yaml Declared 8B Recipe 7 weights, not output quotas mixture.yaml Pipeline Contract · EOS 128009 · uint32 · 01 / Recipe · data_config.yaml Pipeline Contract EOS 128009 · uint32 data_config.yaml Stream Download · Hugging Face → raw JSONL · 02 / Acquire · subprocess Stream Download Hugging Face → raw JSONL subprocess Clean + Deduplicate · quality filters · SHA-256 · 03 / Curate · clean JSONL Clean + Deduplicate quality filters · SHA-256 clean JSONL Resume State · persisted seen set · 03 / Curate · state/ Resume State persisted seen set state/ Tokenize + Pack · BPE + EOS 128009 · 04 / Encode + Pack · shards + manifest Tokenize + Pack BPE + EOS 128009 shards + manifest PretrainDataset · mmap + stitched windows · 05 / Train · one-shard cache PretrainDataset mmap + stitched windows one-shard cache Embedding ID Gap · 128009 > max ID 127999 · 05 / Train · not a working handoff Embedding ID Gap 128009 > max ID 127999 not a working handoff knobs validated YAML source IDs declared recipe raw text JSONL hashes resume metadata clean text accepted documents shards uint32 + manifest out-of-range EOS no ID remap Legend primary data policy / PII async batch data store

Declared recipe ≠ quotas

  • • Web 55% · code 20% · math 10% · arXiv 10% · Cosmopedia 5%.
  • • interleave_sources cycles documents; remaining quota is never decremented.

Unresolved integration gap

  • • TokenStream appends EOS 128009 after validating document tokens.
  • • 128000 rows allow 0–127999. No model/data repair is included.

Source / configured baseline

  • • Revision 3c9675deaf16 · pretrain_a100_502m.yaml
  • • Shared data_config.yaml + mixture.yaml (hashes in RECEIPTS).
  • • Windows may cross EOS; no document attention mask.