unslothai/unsloth · error · HTTPException

dataset_streaming requires hf_dataset; streaming is not supp

Error message

dataset_streaming requires hf_dataset; streaming is not supported for local datasets.

What it means

HTTP 400 when dataset_streaming=true but hf_dataset is empty: streaming only works through the Hugging Face dataset loader, which needs a Hub dataset id. Local datasets have a known length and a file-based loader that cannot stream.

Source

Thrown at studio/backend/routes/training.py:1327

        if request.local_datasets:
            request.local_datasets = _validate_local_dataset_paths(
                request.local_datasets, "Local dataset"
            )
        if request.local_eval_datasets and request.eval_steps > 0:
            request.local_eval_datasets = _validate_local_dataset_paths(
                request.local_eval_datasets, "Local eval dataset"
            )

        from utils.hardware import hardware as _hw
        from utils.hardware import ensure_hardware_detected

        await asyncio.to_thread(ensure_hardware_detected)
        _validate_training_platform(request)

        if request.dataset_streaming:
            if not request.hf_dataset:
                raise HTTPException(
                    status_code = 400,
                    detail = "dataset_streaming requires hf_dataset; streaming is not supported for local datasets.",
                )
            if request.is_dataset_image or request.is_dataset_audio:
                raise HTTPException(
                    status_code = 400,
                    detail = "dataset_streaming is not supported for vision or audio datasets.",
                )
            if request.is_embedding:
                raise HTTPException(
                    status_code = 400,
                    detail = "dataset_streaming is not supported for embedding training; the embedding loader needs the full dataset.",
                )
            if _hw.DEVICE == _hw.DeviceType.MLX:
                raise HTTPException(
                    status_code = 400,
                    detail = "dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.",
                )

View on GitHub (pinned to 203007d190)

Solutions

Set hf_dataset to a Hugging Face dataset id if you truly want streaming
Or disable dataset_streaming to train from the local dataset
For large local datasets, pre-process/shard them instead of expecting streaming

Example fix

// before
{"dataset_streaming": true, "local_datasets": ["/data/train.jsonl"]}  // 400

// after
{"dataset_streaming": false, "local_datasets": ["/data/train.jsonl"]}
// or
{"dataset_streaming": true, "hf_dataset": "org/dataset-name"}

Defensive patterns

Strategy: validation

Validate before calling

def streaming_config_valid(p: dict) -> bool:
    if not p.get("dataset_streaming"):
        return True
    return bool(p.get("hf_dataset"))

Try / catch

resp = client.post("/training/start", payload)
if resp.status_code == 400 and "requires hf_dataset" in resp.text:
    payload["dataset_streaming"] = False  # or set an hf_dataset id
    resp = client.post("/training/start", payload)

Prevention

Remember streaming is HF-Hub-only; never combine it with local dataset paths
Validate the streaming preconditions client-side before posting
Keep one config preset per data source (HF-streamed vs local), not a shared one

When it happens

Trigger: POST /training/start with dataset_streaming: true and no hf_dataset (local_datasets or dataset_local_path supplied instead).

Common situations: Enabling streaming to save memory on a local JSONL/CSV workflow without realizing streaming is Hub-only; config template copied from an HF example applied to local data.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/1f8efb91d676c799. Report an issue: GitHub.