{"record":{"id":"f9a81e5ffcb31232","repo":"unslothai/unsloth","slug":"dataset-streaming-with-evaluation-requires-a-separ","errorCode":null,"errorMessage":"dataset_streaming with evaluation requires a separate eval_split.","messagePattern":"dataset_streaming with evaluation requires a separate eval_split\\.","errorType":"http","errorClass":"HTTPException","httpStatus":422,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1359,"sourceCode":"            if _hw.DEVICE == _hw.DeviceType.MLX:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.\",\n                )\n            if request.max_steps is None or request.max_steps <= 0:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming requires max_steps > 0 because streaming datasets have no known length.\",\n                )\n            if request.train_on_completions:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming is not supported with train_on_completions yet.\",\n                )\n            if request.eval_steps > 0:\n                train_split = request.train_split or \"train\"\n                if not request.eval_split or request.eval_split == train_split:\n                    raise HTTPException(\n                        status_code = 422,\n                        detail = \"dataset_streaming with evaluation requires a separate eval_split.\",\n                    )\n            # Streaming is HF-only: reject when the request also carries a local dataset path or an\n            # S3 config, since those sources cannot be streamed via HF's loader.\n            if request.local_datasets:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = (\n                        \"dataset_streaming is HF-only; remove local_datasets / S3 source. \"\n                        \"Streaming is not supported with local file paths.\"\n                    ),\n                )\n            if request.s3_config is not None:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = (\n                        \"dataset_streaming is HF-only; remove local_datasets / S3 source. \"","sourceCodeStart":1341,"sourceCodeEnd":1377,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1341-L1377","documentation":"HTTP 422 when dataset_streaming=true, eval_steps > 0, but eval_split is missing or equals the train split: streaming evaluation pulls from a separate split stream, so evaluating on the training stream is impossible and must be configured explicitly.","triggerScenarios":"POST /training/start with dataset_streaming: true, eval_steps > 0, and eval_split unset or eval_split == train_split (defaulting train_split to 'train' when unset).","commonSituations":"Config where evaluation used the default split layout on a dataset with only a 'train' split; forgetting eval_split when adding eval_steps to a streaming config.","solutions":["Set eval_split to a real, different split of the dataset (e.g., 'test' or 'validation')","Or set eval_steps to 0 to disable evaluation for this streaming run","Verify the HF dataset actually has the eval split you name"],"exampleFix":"// before\n{\"dataset_streaming\": true, \"eval_steps\": 100}  // 422\n\n// after\n{\"dataset_streaming\": true, \"eval_steps\": 100, \"train_split\": \"train\", \"eval_split\": \"validation\"}","handlingStrategy":"validation","validationCode":"def streaming_config_valid(p: dict) -> bool:\n    if not p.get(\"dataset_streaming\") or not (p.get(\"eval_steps\", 0) or 0) > 0:\n        return True\n    train_split = p.get(\"train_split\") or \"train\"\n    eval_split = p.get(\"eval_split\")\n    return bool(eval_split) and eval_split != train_split","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 422 and \"eval_split\" in resp.text:\n    payload[\"eval_split\"] = \"validation\"  # must differ from train_split and exist in the dataset\n    resp = client.post(\"/training/start\", payload)","preventionTips":["Set an explicit eval_split whenever eval_steps > 0 on streaming runs","Confirm the HF dataset actually contains the eval split you name","Default train_split is 'train': never set eval_split to 'train'"],"tags":["training","dataset-streaming","evaluation","split","http-422"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}