{"record":{"id":"273dc862918e2b07","repo":"hiyouga/LlamaFactory","slug":"turn-off-streaming-when-saving-dataset-to-disk","errorCode":null,"errorMessage":"Turn off `streaming` when saving dataset to disk.","messagePattern":"Turn off `streaming` when saving dataset to disk\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/data/loader.py","lineNumber":299,"sourceCode":"    stage: Literal[\"pt\", \"sft\", \"rm\", \"ppo\", \"kto\"],\n    tokenizer: \"PreTrainedTokenizer\",\n    processor: Optional[\"ProcessorMixin\"] = None,\n) -> \"DatasetModule\":\n    r\"\"\"Get the train dataset and optionally gets the evaluation dataset.\"\"\"\n    # Load tokenized dataset if path exists\n    if data_args.tokenized_path is not None:\n        if has_tokenized_data(data_args.tokenized_path):\n            logger.warning_rank0(\"Loading dataset from disk will ignore other data arguments.\")\n            tokenized_data = load_from_disk(data_args.tokenized_path)\n            dataset_module = get_dataset_module(tokenized_data)\n            if data_args.streaming:\n                dataset_module[\"train_dataset\"] = dataset_module[\"train_dataset\"].to_iterable_dataset()\n\n            logger.info_rank0(f\"Loaded tokenized dataset from {data_args.tokenized_path}.\")\n            return dataset_module\n\n        if data_args.streaming:\n            raise ValueError(\"Turn off `streaming` when saving dataset to disk.\")\n\n    # Load and preprocess dataset\n    with training_args.main_process_first(desc=\"load dataset\", local=(not data_args.data_shared_file_system)):\n        dataset = _get_merged_dataset(data_args.dataset, model_args, data_args, training_args, stage)\n        eval_dataset = _get_merged_dataset(\n            data_args.eval_dataset,\n            model_args,\n            data_args,\n            training_args,\n            stage,\n            return_dict=data_args.eval_on_each_dataset,\n        )\n\n    with training_args.main_process_first(desc=\"pre-process dataset\", local=(not data_args.data_shared_file_system)):\n        # move front to make sure eval_dataset(if contain or split) can preprocessed appropriately\n        train_dict, eval_dict = split_dataset(dataset, eval_dataset, data_args, seed=training_args.seed)\n\n        if \"train\" in train_dict:","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/loader.py#L281-L317","documentation":"ValueError raised in get_dataset_module when data_args.streaming is enabled together with a tokenized_path: you cannot lazily stream a dataset while also asking LlamaFactory to save the tokenized result to disk. The check fires only when tokenized_path is set and no prior tokenized data exists there.","triggerScenarios":"A training YAML containing both streaming: true and tokenized_path: some/dir, with no previously saved tokenized dataset at that path. The guard prevents an incoherent save-while-streaming request.","commonSituations":"Reusing a streaming config (used for huge hub datasets) and later adding tokenized_path to cache preprocessing; flipping streaming on to avoid full download while forgetting the save option is still set.","solutions":["Set streaming: false in the YAML when you want tokenized_path to be written.","If you only want to load previously tokenized data, keep tokenized_path and drop other data args; streaming is then applied automatically after loading from disk.","Remove tokenized_path if you truly need streaming and do not need the on-disk tokenized cache."],"exampleFix":"# before (train.yaml)\nstreaming: true\ntokenized_path: data/tokenized\n\n# after\nstreaming: false\ntokenized_path: data/tokenized","handlingStrategy":"validation","validationCode":"def streaming_save_compatible(streaming: bool, tokenized_path: str | None) -> bool:\n    return not (streaming and tokenized_path is not None)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat streaming and tokenized_path as mutually exclusive in YAML templates.","Comment YAML keys you disable so stale flags do not linger between runs."],"tags":["streaming","tokenized-path","config","dataset"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}