{"record":{"id":"1f8efb91d676c799","repo":"unslothai/unsloth","slug":"dataset-streaming-requires-hf-dataset-streaming-i","errorCode":null,"errorMessage":"dataset_streaming requires hf_dataset; streaming is not supported for local datasets.","messagePattern":"dataset_streaming requires hf_dataset; streaming is not supported for local datasets\\.","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1327,"sourceCode":"\n        if request.local_datasets:\n            request.local_datasets = _validate_local_dataset_paths(\n                request.local_datasets, \"Local dataset\"\n            )\n        if request.local_eval_datasets and request.eval_steps > 0:\n            request.local_eval_datasets = _validate_local_dataset_paths(\n                request.local_eval_datasets, \"Local eval dataset\"\n            )\n\n        from utils.hardware import hardware as _hw\n        from utils.hardware import ensure_hardware_detected\n\n        await asyncio.to_thread(ensure_hardware_detected)\n        _validate_training_platform(request)\n\n        if request.dataset_streaming:\n            if not request.hf_dataset:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming requires hf_dataset; streaming is not supported for local datasets.\",\n                )\n            if request.is_dataset_image or request.is_dataset_audio:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for vision or audio datasets.\",\n                )\n            if request.is_embedding:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset.\",\n                )\n            if _hw.DEVICE == _hw.DeviceType.MLX:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.\",\n                )","sourceCodeStart":1309,"sourceCodeEnd":1345,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1309-L1345","documentation":"HTTP 400 when dataset_streaming=true but hf_dataset is empty: streaming only works through the Hugging Face dataset loader, which needs a Hub dataset id. Local datasets have a known length and a file-based loader that cannot stream.","triggerScenarios":"POST /training/start with dataset_streaming: true and no hf_dataset (local_datasets or dataset_local_path supplied instead).","commonSituations":"Enabling streaming to save memory on a local JSONL/CSV workflow without realizing streaming is Hub-only; config template copied from an HF example applied to local data.","solutions":["Set hf_dataset to a Hugging Face dataset id if you truly want streaming","Or disable dataset_streaming to train from the local dataset","For large local datasets, pre-process/shard them instead of expecting streaming"],"exampleFix":"// before\n{\"dataset_streaming\": true, \"local_datasets\": [\"/data/train.jsonl\"]}  // 400\n\n// after\n{\"dataset_streaming\": false, \"local_datasets\": [\"/data/train.jsonl\"]}\n// or\n{\"dataset_streaming\": true, \"hf_dataset\": \"org/dataset-name\"}","handlingStrategy":"validation","validationCode":"def streaming_config_valid(p: dict) -> bool:\n    if not p.get(\"dataset_streaming\"):\n        return True\n    return bool(p.get(\"hf_dataset\"))","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 400 and \"requires hf_dataset\" in resp.text:\n    payload[\"dataset_streaming\"] = False  # or set an hf_dataset id\n    resp = client.post(\"/training/start\", payload)","preventionTips":["Remember streaming is HF-Hub-only; never combine it with local dataset paths","Validate the streaming preconditions client-side before posting","Keep one config preset per data source (HF-streamed vs local), not a shared one"],"tags":["training","dataset-streaming","validation","http-400"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}