{"record":{"id":"4a5ab17c4c4d3332","repo":"unslothai/unsloth","slug":"dataset-streaming-requires-max-steps-0-because-s","errorCode":null,"errorMessage":"dataset_streaming requires max_steps > 0 because streaming datasets have no known length.","messagePattern":"dataset_streaming requires max_steps > 0 because streaming datasets have no known length\\.","errorType":"http","errorClass":"HTTPException","httpStatus":422,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1347,"sourceCode":"                    detail = \"dataset_streaming requires hf_dataset; streaming is not supported for local datasets.\",\n                )\n            if request.is_dataset_image or request.is_dataset_audio:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for vision or audio datasets.\",\n                )\n            if request.is_embedding:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset.\",\n                )\n            if _hw.DEVICE == _hw.DeviceType.MLX:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.\",\n                )\n            if request.max_steps is None or request.max_steps <= 0:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming requires max_steps > 0 because streaming datasets have no known length.\",\n                )\n            if request.train_on_completions:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming is not supported with train_on_completions yet.\",\n                )\n            if request.eval_steps > 0:\n                train_split = request.train_split or \"train\"\n                if not request.eval_split or request.eval_split == train_split:\n                    raise HTTPException(\n                        status_code = 422,\n                        detail = \"dataset_streaming with evaluation requires a separate eval_split.\",\n                    )\n            # Streaming is HF-only: reject when the request also carries a local dataset path or an\n            # S3 config, since those sources cannot be streamed via HF's loader.\n            if request.local_datasets:","sourceCodeStart":1329,"sourceCodeEnd":1365,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1329-L1365","documentation":"HTTP 422 when dataset_streaming=true and max_steps is None or <= 0: a streaming dataset has no known length, so epoch-based stopping cannot work and an explicit step budget is mandatory.","triggerScenarios":"POST /training/start with dataset_streaming: true without a positive max_steps (relying on num_train_epochs / default stopping).","commonSituations":"Porting a config that used epochs on a bounded dataset to a streamed dataset; omitting max_steps because non-streaming runs default sensibly.","solutions":["Set max_steps to a positive integer equal to the desired training budget (steps = epochs x approx_stream_length)","Or disable streaming so length-based epoch training applies","Sanity-check max_steps against your desired effective epoch count for the streamed data"],"exampleFix":"// before\n{\"dataset_streaming\": true, \"hf_dataset\": \"org/ds\"}  // 422\n\n// after\n{\"dataset_streaming\": true, \"hf_dataset\": \"org/ds\", \"max_steps\": 5000}","handlingStrategy":"validation","validationCode":"def streaming_config_valid(p: dict) -> bool:\n    if not p.get(\"dataset_streaming\"):\n        return True\n    ms = p.get(\"max_steps\")\n    return isinstance(ms, int) and ms > 0","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 422 and \"max_steps\" in resp.text:\n    payload[\"max_steps\"] = estimate_steps(desired_epochs, approx_stream_length)\n    resp = client.post(\"/training/start\", payload)","preventionTips":["Always set an explicit positive max_steps for streaming runs","Convert epoch budgets to step budgets when switching to streaming","Remember streamed datasets have no known length: epoch semantics do not apply"],"tags":["training","dataset-streaming","max-steps","http-422","validation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}