hiyouga/LlamaFactory · error · ValueError

Turn off `streaming` when saving dataset to disk.

Error message

Turn off `streaming` when saving dataset to disk.

What it means

ValueError raised in get_dataset_module when data_args.streaming is enabled together with a tokenized_path: you cannot lazily stream a dataset while also asking LlamaFactory to save the tokenized result to disk. The check fires only when tokenized_path is set and no prior tokenized data exists there.

Source

Thrown at src/llamafactory/data/loader.py:299

    stage: Literal["pt", "sft", "rm", "ppo", "kto"],
    tokenizer: "PreTrainedTokenizer",
    processor: Optional["ProcessorMixin"] = None,
) -> "DatasetModule":
    r"""Get the train dataset and optionally gets the evaluation dataset."""
    # Load tokenized dataset if path exists
    if data_args.tokenized_path is not None:
        if has_tokenized_data(data_args.tokenized_path):
            logger.warning_rank0("Loading dataset from disk will ignore other data arguments.")
            tokenized_data = load_from_disk(data_args.tokenized_path)
            dataset_module = get_dataset_module(tokenized_data)
            if data_args.streaming:
                dataset_module["train_dataset"] = dataset_module["train_dataset"].to_iterable_dataset()

            logger.info_rank0(f"Loaded tokenized dataset from {data_args.tokenized_path}.")
            return dataset_module

        if data_args.streaming:
            raise ValueError("Turn off `streaming` when saving dataset to disk.")

    # Load and preprocess dataset
    with training_args.main_process_first(desc="load dataset", local=(not data_args.data_shared_file_system)):
        dataset = _get_merged_dataset(data_args.dataset, model_args, data_args, training_args, stage)
        eval_dataset = _get_merged_dataset(
            data_args.eval_dataset,
            model_args,
            data_args,
            training_args,
            stage,
            return_dict=data_args.eval_on_each_dataset,
        )

    with training_args.main_process_first(desc="pre-process dataset", local=(not data_args.data_shared_file_system)):
        # move front to make sure eval_dataset(if contain or split) can preprocessed appropriately
        train_dict, eval_dict = split_dataset(dataset, eval_dataset, data_args, seed=training_args.seed)

        if "train" in train_dict:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set streaming: false in the YAML when you want tokenized_path to be written.
2. If you only want to load previously tokenized data, keep tokenized_path and drop other data args; streaming is then applied automatically after loading from disk.
3. Remove tokenized_path if you truly need streaming and do not need the on-disk tokenized cache.

Example fix

# before (train.yaml)
streaming: true
tokenized_path: data/tokenized

# after
streaming: false
tokenized_path: data/tokenized

def streaming_save_compatible(streaming: bool, tokenized_path: str | None) -> bool:
    return not (streaming and tokenized_path is not None)

When it happens

Trigger: A training YAML containing both streaming: true and tokenized_path: some/dir, with no previously saved tokenized dataset at that path. The guard prevents an incoherent save-while-streaming request.

Common situations: Reusing a streaming config (used for huge hub datasets) and later adding tokenized_path to cache preprocessing; flipping streaming on to avoid full download while forgetting the save option is still set.

Related errors

• The model does not have a submodule named '{submodule_name}'
• Unsupported model type: {getattr(config, 'model_type')}.
• Dataset converter {name} not found.
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `eval_dataset` is not None.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/273dc862918e2b07. Report an issue: GitHub.