{"record":{"id":"b9df33ffb55de260","repo":"hiyouga/LlamaFactory","slug":"all-datasets-must-be-streaming-or-non-streaming","errorCode":null,"errorMessage":"All datasets must be streaming or non-streaming.","messagePattern":"All datasets must be streaming or non-streaming\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/data_engine.py","lineNumber":88,"sourceCode":"    def _get_dataset_info(self) -> None:\n        \"\"\"Get dataset info from data arguments.\"\"\"\n        if self.path.endswith(\".yaml\") and os.path.isfile(self.path):  # local file\n            self.dataset_infos = OmegaConf.load(self.path)\n        elif self.path.endswith(\".yaml\"):  # hf hub uri, e.g. llamafactory/v1-sft-demo/dataset_info.yaml\n            repo_id, filename = os.path.split(self.path)\n            filepath = hf_hub_download(repo_id=repo_id, filename=filename, repo_type=\"dataset\")\n            self.dataset_infos = OmegaConf.load(filepath)\n        elif os.path.exists(self.path):  # local file(s)\n            self.dataset_infos = {\"default\": {\"path\": self.path, \"source\": \"local\"}}\n        else:  # hf hub dataset, e.g. llamafactory/v1-sft-demo\n            self.dataset_infos = {\"default\": {\"path\": self.path}}\n\n    def _load_dataset(self) -> None:\n        \"\"\"Load datasets according to dataset info.\"\"\"\n        is_streaming = [dataset_info.get(\"streaming\", False) for dataset_info in self.dataset_infos.values()]\n        self.streaming = any(is_streaming)\n        if all(is_streaming) != any(is_streaming):\n            raise ValueError(\"All datasets must be streaming or non-streaming.\")\n\n        for dataset_name, dataset_info in self.dataset_infos.items():\n            split = dataset_info.get(\"split\", \"train\")\n            if dataset_info.get(\"source\", \"hf_hub\") == \"hf_hub\":\n                from datasets import load_dataset\n\n                self.datasets[dataset_name] = load_dataset(dataset_info[\"path\"], split=split, streaming=self.streaming)\n            else:  # data loader plugin\n                from ..plugins.data_plugins.loader import DataLoaderPlugin\n\n                self.datasets[dataset_name] = DataLoaderPlugin(dataset_info[\"source\"]).load(dataset_info)\n\n    def _build_data_index(self) -> None:\n        \"\"\"Build dataset index.\n\n        Multi-turn SFT conversations are prefix-expanded: one index entry per supervised assistant\n        turn, so ``len()`` reflects the true number of training samples (each trained on its last\n        turn). Entries are ``(dataset_name, sample_index, cut)``; ``cut`` is the prefix length","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/data_engine.py#L70-L106","documentation":"The v1 `DataEngine` loads every configured dataset and requires them to be homogeneous in streaming mode: it computes `is_streaming` per dataset and raises if some are streaming and some are not (`all(...) != any(...)`). Mixed mode is rejected because a single `self.streaming` flag drives loading and iteration for the whole engine.","triggerScenarios":"A dataset_info config where one entry has `streaming: true` and another defaults to false — e.g. mixing a streamed HF hub dataset with a local parquet/jsonl file in the same `dataset` list.","commonSituations":"Prototyping with a small local dataset plus a large streamed dataset to save disk; partial config edits that added `streaming: true` to only one dataset entry.","solutions":["Make all entries consistent: add `streaming: true` to every dataset, or remove it from all","Prefer non-streaming for small/local datasets by downloading the streamed one first (`load_dataset(..., download_mode)` or hub snapshot) and pointing both at local files","Check the rendered `dataset_infos` (OmegaConf merge result) to spot the divergent entry"],"exampleFix":"# before (yaml dataset_info)\ndataset1:\n  path: huge/hub-dataset\n  streaming: true\ndataset2:\n  path: data/local.jsonl   # non-streaming -> mixed\n\n# after (yaml dataset_info)\ndataset1:\n  path: huge/hub-dataset\ndataset2:\n  path: data/local.jsonl   # download hub dataset locally instead, both non-streaming","handlingStrategy":"validation","validationCode":"def validate_streaming(dataset_infos: dict) -> None:\n    flags = [bool(v.get(\"streaming\", False)) for v in dataset_infos.values()]\n    if any(flags) and not all(flags):\n        raise SystemExit(f\"mixed streaming flags: {dict(zip(dataset_infos, flags))}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set streaming per-engine, never per-dataset when mixing sources","Materialize small datasets locally instead of mixing stream modes","Lint dataset_info YAML so all entries share the same streaming value"],"tags":["v1","datasets","streaming","configuration"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}