hiyouga/LlamaFactory · error · ValueError

Cannot specify `val_size` if `eval_dataset` is not None.

Error message

Cannot specify `val_size` if `eval_dataset` is not None.

What it means

val_size splits the training dataset into train/val parts, which is redundant and ambiguous when an explicit eval_dataset is already provided. DataArguments.post_init raises ValueError when eval_dataset is not None and val_size > 1e-6, forcing you to choose one validation source.

Source

Thrown at src/llamafactory/hparams/data_args.py:163

    )

    def __post_init__(self):
        def split_arg(arg):
            if isinstance(arg, str):
                return [item.strip() for item in arg.split(",")]
            return arg

        self.dataset = split_arg(self.dataset)
        self.eval_dataset = split_arg(self.eval_dataset)

        if self.media_dir is None:
            self.media_dir = self.dataset_dir

        if self.dataset is None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `dataset` is None.")

        if self.eval_dataset is not None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `eval_dataset` is not None.")

        if self.interleave_probs is not None:
            if self.mix_strategy == "concat":
                raise ValueError("`interleave_probs` is only valid for interleaved mixing.")

            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))
            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):
                raise ValueError("The length of dataset and interleave probs should be identical.")

            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):
                raise ValueError("The length of eval dataset and interleave probs should be identical.")

        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:
            raise ValueError("Streaming mode should have an integer val size.")

        if self.streaming and self.max_samples is not None:
            raise ValueError("`max_samples` is incompatible with `streaming`.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Delete val_size (or set 0) and keep eval_dataset for validation.
2. Or remove eval_dataset and keep val_size if you want a random split from training data.

Example fix

# before
dataset: alpaca_demo
eval_dataset: mmlu_demo
val_size: 0.1

# after
dataset: alpaca_demo
eval_dataset: mmlu_demo

assert not (data_args.eval_dataset is not None and data_args.val_size > 1e-6), \
    "choose eval_dataset OR val_size, not both"

When it happens

Trigger: YAML with both eval_dataset: <name> and val_size: 0.05; dataset_info.json defining evaluation data while the config still carries a val_size fraction from an earlier setup.

Common situations: Iterating on configs: first using val_size for a quick split, later adding a proper eval_dataset but forgetting val_size; merging config snippets from examples that each used a different validation approach.

Related errors

• Cannot specify `val_size` if `dataset` is None.
• `interleave_probs` is only valid for interleaved mixing.
• The length of eval dataset and interleave probs should be id
• Streaming mode should have an integer val size.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c8e0fb15fe08feff. Report an issue: GitHub.