What causes "All datasets must be streaming or non-streaming." in hiyouga/LlamaFactory?

Make all entries consistent: add `streaming: true` to every dataset, or remove it from all

hiyouga/LlamaFactory · error · ValueError

All datasets must be streaming or non-streaming.

Error message

All datasets must be streaming or non-streaming.

What it means

The v1 `DataEngine` loads every configured dataset and requires them to be homogeneous in streaming mode: it computes `is_streaming` per dataset and raises if some are streaming and some are not (`all(...) != any(...)`). Mixed mode is rejected because a single `self.streaming` flag drives loading and iteration for the whole engine.

Source

Thrown at src/llamafactory/v1/core/data_engine.py:88

    def _get_dataset_info(self) -> None:
        """Get dataset info from data arguments."""
        if self.path.endswith(".yaml") and os.path.isfile(self.path):  # local file
            self.dataset_infos = OmegaConf.load(self.path)
        elif self.path.endswith(".yaml"):  # hf hub uri, e.g. llamafactory/v1-sft-demo/dataset_info.yaml
            repo_id, filename = os.path.split(self.path)
            filepath = hf_hub_download(repo_id=repo_id, filename=filename, repo_type="dataset")
            self.dataset_infos = OmegaConf.load(filepath)
        elif os.path.exists(self.path):  # local file(s)
            self.dataset_infos = {"default": {"path": self.path, "source": "local"}}
        else:  # hf hub dataset, e.g. llamafactory/v1-sft-demo
            self.dataset_infos = {"default": {"path": self.path}}

    def _load_dataset(self) -> None:
        """Load datasets according to dataset info."""
        is_streaming = [dataset_info.get("streaming", False) for dataset_info in self.dataset_infos.values()]
        self.streaming = any(is_streaming)
        if all(is_streaming) != any(is_streaming):
            raise ValueError("All datasets must be streaming or non-streaming.")

        for dataset_name, dataset_info in self.dataset_infos.items():
            split = dataset_info.get("split", "train")
            if dataset_info.get("source", "hf_hub") == "hf_hub":
                from datasets import load_dataset

                self.datasets[dataset_name] = load_dataset(dataset_info["path"], split=split, streaming=self.streaming)
            else:  # data loader plugin
                from ..plugins.data_plugins.loader import DataLoaderPlugin

                self.datasets[dataset_name] = DataLoaderPlugin(dataset_info["source"]).load(dataset_info)

    def _build_data_index(self) -> None:
        """Build dataset index.

        Multi-turn SFT conversations are prefix-expanded: one index entry per supervised assistant
        turn, so ``len()`` reflects the true number of training samples (each trained on its last
        turn). Entries are ``(dataset_name, sample_index, cut)``; ``cut`` is the prefix length

View on GitHub (pinned to f28afaf635)

Solutions

Make all entries consistent: add `streaming: true` to every dataset, or remove it from all
Prefer non-streaming for small/local datasets by downloading the streamed one first (`load_dataset(..., download_mode)` or hub snapshot) and pointing both at local files
Check the rendered `dataset_infos` (OmegaConf merge result) to spot the divergent entry

Example fix

# before (yaml dataset_info)
dataset1:
  path: huge/hub-dataset
  streaming: true
dataset2:
  path: data/local.jsonl   # non-streaming -> mixed

# after (yaml dataset_info)
dataset1:
  path: huge/hub-dataset
dataset2:
  path: data/local.jsonl   # download hub dataset locally instead, both non-streaming

Defensive patterns

Strategy: validation

Validate before calling

def validate_streaming(dataset_infos: dict) -> None:
    flags = [bool(v.get("streaming", False)) for v in dataset_infos.values()]
    if any(flags) and not all(flags):
        raise SystemExit(f"mixed streaming flags: {dict(zip(dataset_infos, flags))}")

Prevention

Set streaming per-engine, never per-dataset when mixing sources
Materialize small datasets locally instead of mixing stream modes
Lint dataset_info YAML so all entries share the same streaming value

When it happens

Trigger: A dataset_info config where one entry has `streaming: true` and another defaults to false — e.g. mixing a streamed HF hub dataset with a local parquet/jsonl file in the same `dataset` list.

Common situations: Prototyping with a small local dataset plus a large streamed dataset to save disk; partial config edits that added `streaming: true` to only one dataset entry.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/b9df33ffb55de260. Report an issue: GitHub.