hiyouga/LlamaFactory · error · ValueError

Cannot specify `val_size` if `dataset` is None.

Error message

Cannot specify `val_size` if `dataset` is None.

What it means

DataArguments post-init validates that val_size (a float: fraction if <1, absolute count if >=1) is only meaningful when a training dataset exists. If dataset is None but val_size > 1e-6, it raises ValueError — you cannot carve a validation split out of nothing.

Source

Thrown at src/llamafactory/hparams/data_args.py:160

    data_shared_file_system: bool = field(
        default=False,
        metadata={"help": "Whether or not to use a shared file system for the datasets."},
    )

    def __post_init__(self):
        def split_arg(arg):
            if isinstance(arg, str):
                return [item.strip() for item in arg.split(",")]
            return arg

        self.dataset = split_arg(self.dataset)
        self.eval_dataset = split_arg(self.eval_dataset)

        if self.media_dir is None:
            self.media_dir = self.dataset_dir

        if self.dataset is None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `dataset` is None.")

        if self.eval_dataset is not None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `eval_dataset` is not None.")

        if self.interleave_probs is not None:
            if self.mix_strategy == "concat":
                raise ValueError("`interleave_probs` is only valid for interleaved mixing.")

            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))
            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):
                raise ValueError("The length of dataset and interleave probs should be identical.")

            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):
                raise ValueError("The length of eval dataset and interleave probs should be identical.")

        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:
            raise ValueError("Streaming mode should have an integer val size.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove val_size from the config (or set val_size: 0) when no training dataset is used.
2. If you intended to train, add the dataset (and its dataset_info.json entry).
3. If you only want evaluation, supply eval_dataset and drop val_size entirely.

Example fix

# before
### dataset
dataset: null
val_size: 0.1

# after
### dataset
dataset: alpaca_demo
val_size: 0.1

assert not (data_args.dataset is None and data_args.val_size > 1e-6), \
    "val_size requires a training dataset"

When it happens

Trigger: A YAML config with eval_dataset unset, dataset unset (e.g. a pure-eval or chat entry misused for training), but val_size: 0.1 still present; leftover val_size from a template config after removing dataset.

Common situations: Copying a training YAML and deleting the dataset lines for a quick test run; misconfigured webui exports that keep val_size with no dataset; switching to eval-only workflows without cleaning val_size.

Related errors

• Cannot specify `val_size` if `eval_dataset` is not None.
• `interleave_probs` is only valid for interleaved mixing.
• The length of eval dataset and interleave probs should be id
• Streaming mode should have an integer val size.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/44c58802af14e7d2. Report an issue: GitHub.