{"record":{"id":"48184289fb3f5b91","repo":"unslothai/unsloth","slug":"dataset-streaming-is-not-supported-for-embedding-t","errorCode":null,"errorMessage":"dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset.","messagePattern":"dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset\\.","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"studio/backend/routes/training.py","lineNumber":1337,"sourceCode":"        from utils.hardware import hardware as _hw\n        from utils.hardware import ensure_hardware_detected\n\n        await asyncio.to_thread(ensure_hardware_detected)\n        _validate_training_platform(request)\n\n        if request.dataset_streaming:\n            if not request.hf_dataset:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming requires hf_dataset; streaming is not supported for local datasets.\",\n                )\n            if request.is_dataset_image or request.is_dataset_audio:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for vision or audio datasets.\",\n                )\n            if request.is_embedding:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset.\",\n                )\n            if _hw.DEVICE == _hw.DeviceType.MLX:\n                raise HTTPException(\n                    status_code = 400,\n                    detail = \"dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.\",\n                )\n            if request.max_steps is None or request.max_steps <= 0:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming requires max_steps > 0 because streaming datasets have no known length.\",\n                )\n            if request.train_on_completions:\n                raise HTTPException(\n                    status_code = 422,\n                    detail = \"dataset_streaming is not supported with train_on_completions yet.\",\n                )","sourceCodeStart":1319,"sourceCodeEnd":1355,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/routes/training.py#L1319-L1355","documentation":"HTTP 400 when dataset_streaming=true and is_embedding is set: the embedding training loader requires the full dataset in memory (it needs complete pass structure), which conflicts with streaming's lazy iteration.","triggerScenarios":"POST /training/start with dataset_streaming: true and is_embedding: true.","commonSituations":"Applying a streaming config preset to embedding fine-tuning; assuming streaming works uniformly across all training modes.","solutions":["Disable dataset_streaming for embedding training","Reduce dataset size or use a machine with more RAM for large embedding datasets","Reserve streaming for standard text fine-tuning runs only"],"exampleFix":"// before\n{\"dataset_streaming\": true, \"is_embedding\": true}  // 400\n\n// after\n{\"dataset_streaming\": false, \"is_embedding\": true}","handlingStrategy":"validation","validationCode":"def streaming_config_valid(p: dict) -> bool:\n    if not p.get(\"dataset_streaming\"):\n        return True\n    return not p.get(\"is_embedding\")","typeGuard":null,"tryCatchPattern":"resp = client.post(\"/training/start\", payload)\nif resp.status_code == 400 and \"embedding\" in resp.text:\n    payload[\"dataset_streaming\"] = False\n    resp = client.post(\"/training/start\", payload)","preventionTips":["Never enable streaming for embedding training runs","Budget RAM for the full dataset when training embeddings","Encode the embedding/streaming exclusion in your config schema validation"],"tags":["training","dataset-streaming","embedding","http-400"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}