unslothai/unsloth · error · HTTPException

dataset_streaming with evaluation requires a separate eval_s

Error message

dataset_streaming with evaluation requires a separate eval_split.

What it means

HTTP 422 when dataset_streaming=true, eval_steps > 0, but eval_split is missing or equals the train split: streaming evaluation pulls from a separate split stream, so evaluating on the training stream is impossible and must be configured explicitly.

Source

Thrown at studio/backend/routes/training.py:1359

            if _hw.DEVICE == _hw.DeviceType.MLX:
                raise HTTPException(
                    status_code = 400,
                    detail = "dataset_streaming is not yet supported on Apple Silicon (MLX); the MLX loader materializes the full dataset.",
                )
            if request.max_steps is None or request.max_steps <= 0:
                raise HTTPException(
                    status_code = 422,
                    detail = "dataset_streaming requires max_steps > 0 because streaming datasets have no known length.",
                )
            if request.train_on_completions:
                raise HTTPException(
                    status_code = 422,
                    detail = "dataset_streaming is not supported with train_on_completions yet.",
                )
            if request.eval_steps > 0:
                train_split = request.train_split or "train"
                if not request.eval_split or request.eval_split == train_split:
                    raise HTTPException(
                        status_code = 422,
                        detail = "dataset_streaming with evaluation requires a separate eval_split.",
                    )
            # Streaming is HF-only: reject when the request also carries a local dataset path or an
            # S3 config, since those sources cannot be streamed via HF's loader.
            if request.local_datasets:
                raise HTTPException(
                    status_code = 400,
                    detail = (
                        "dataset_streaming is HF-only; remove local_datasets / S3 source. "
                        "Streaming is not supported with local file paths."
                    ),
                )
            if request.s3_config is not None:
                raise HTTPException(
                    status_code = 400,
                    detail = (
                        "dataset_streaming is HF-only; remove local_datasets / S3 source. "

View on GitHub (pinned to 203007d190)

Solutions

Set eval_split to a real, different split of the dataset (e.g., 'test' or 'validation')
Or set eval_steps to 0 to disable evaluation for this streaming run
Verify the HF dataset actually has the eval split you name

Example fix

// before
{"dataset_streaming": true, "eval_steps": 100}  // 422

// after
{"dataset_streaming": true, "eval_steps": 100, "train_split": "train", "eval_split": "validation"}

Defensive patterns

Strategy: validation

Validate before calling

def streaming_config_valid(p: dict) -> bool:
    if not p.get("dataset_streaming") or not (p.get("eval_steps", 0) or 0) > 0:
        return True
    train_split = p.get("train_split") or "train"
    eval_split = p.get("eval_split")
    return bool(eval_split) and eval_split != train_split

Try / catch

resp = client.post("/training/start", payload)
if resp.status_code == 422 and "eval_split" in resp.text:
    payload["eval_split"] = "validation"  # must differ from train_split and exist in the dataset
    resp = client.post("/training/start", payload)

Prevention

Set an explicit eval_split whenever eval_steps > 0 on streaming runs
Confirm the HF dataset actually contains the eval split you name
Default train_split is 'train': never set eval_split to 'train'

When it happens

Trigger: POST /training/start with dataset_streaming: true, eval_steps > 0, and eval_split unset or eval_split == train_split (defaulting train_split to 'train' when unset).

Common situations: Config where evaluation used the default split layout on a dataset with only a 'train' split; forgetting eval_split when adding eval_steps to a streaming config.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/f9a81e5ffcb31232. Report an issue: GitHub.