{"record":{"id":"b185c6602c5a5e89","repo":"karpathy/nanochat","slug":"unknown-dataset-tag-dataset-tag","errorCode":null,"errorMessage":"Unknown dataset tag: {dataset_tag}","messagePattern":"Unknown dataset tag: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"dev/repackage_data_reference.py","lineNumber":63,"sourceCode":"    }\n    output_dirname = \"fineweb_edu\"\n    data_column_name = \"text\"\n    tokenizer = None\n    upload_tag = \"fineweb-edu-100b-shuffle\"\n\nelif dataset_tag == \"climbmix\":\n    import tiktoken # the ClimbMix data is stored tokenized with GPT-2 tokenizer\n    dataset_kwargs = {\n        \"path\": \"nvidia/Nemotron-ClimbMix\",\n        \"split\": \"train\",\n    }\n    output_dirname = \"climbmix\"\n    data_column_name = \"tokens\"\n    tokenizer = tiktoken.encoding_for_model(\"gpt-2\")\n    upload_tag = \"climbmix-400b-shuffle\"\n\nelse:\n    raise ValueError(f\"Unknown dataset tag: {dataset_tag}\")\n\n# Source dataset\nds = load_dataset(**dataset_kwargs)\n\n# Shuffle to scramble the order\nds = ds.shuffle(seed=42)\nndocs = len(ds) # total number of documents to process\nprint(f\"Total number of documents: {ndocs}\")\n\n# Repackage into parquet files\noutput_dir = f\"/home/ubuntu/.cache/nanochat/base_data_{output_dirname}\"\nos.makedirs(output_dir, exist_ok=True)\n\n# Write to parquet files\nchars_per_shard = 250_000_000\nrow_group_size = 1024 # HF uses 1000 but we use multiple of 2, nicer for distributed data loader later\nshard_docs = []\nshard_index = 0","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/karpathy/nanochat/blob/92d63d4e8bb4df75c3b71618f31ddde2378b2bcd/dev/repackage_data_reference.py#L45-L81","documentation":"dev/repackage_data_reference.py validates the module-level `dataset_tag` variable against the only two supported values: 'fineweb_edu' and 'climbmix'. Any other string falls through the if/elif chain to the else branch and raises ValueError. The script is documentation-only (it describes how the two hosted datasets were prepared), so this is a guard against editing the tag to a dataset that has no preparation recipe.","triggerScenarios":"Setting `dataset_tag` at line 36 to anything other than the literal strings 'fineweb_edu' or 'climbmix' (e.g. 'fineweb', 'ClimbMix' with different casing, or a new dataset name) before running the script.","commonSituations":"Developers copying this reference script to prepare a new dataset and changing only the tag without adding a matching elif branch; typos or casing mismatches ('FineWebEdu', 'climb_mix').","solutions":["Set dataset_tag to one of the supported literals: 'fineweb_edu' or 'climbmix'.","If you need a new dataset, add a new `elif dataset_tag == \"<your_tag>\":` block that fills in dataset_kwargs, output_dirname, data_column_name, tokenizer, and upload_tag before the else branch.","Check for typos/casing — the comparison is exact and case-sensitive."],"exampleFix":"// before\ndataset_tag = \"fineweb\"\n\n// after\ndataset_tag = \"fineweb_edu\"  # or \"climbmix\"","handlingStrategy":"validation","validationCode":"SUPPORTED_DATASET_TAGS = {\"fineweb_edu\", \"climbmix\"}\nassert dataset_tag in SUPPORTED_DATASET_TAGS, f\"dataset_tag must be one of {SUPPORTED_DATASET_TAGS}, got {dataset_tag!r}\"","typeGuard":null,"tryCatchPattern":"try:\n    run_repackage(dataset_tag)\nexcept ValueError as e:\n    if \"Unknown dataset tag\" in str(e):\n        raise SystemExit(f\"Unsupported dataset tag {dataset_tag!r}; supported: fineweb_edu, climbmix\")\n    raise","preventionTips":["Define supported tags as a module-level constant set and validate before the if/elif chain.","Keep tag names lowercase with underscores and match the literal strings exactly."],"tags":["nanochat","dataset","configuration","validation"],"backgroundTag":null,"analyzedSha":"92d63d4e8bb4df75c3b71618f31ddde2378b2bcd","analyzedAt":"2026-08-15T03:11:54.371Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}