hiyouga/LlamaFactory · error · ValueError

Cannot specify `val_size` if `eval_dataset` is not None.

Error message

Cannot specify `val_size` if `eval_dataset` is not None.

What it means

split_dataset raises ValueError when an explicit eval_dataset is provided AND data_args.val_size > 1e-6. The two are mutually exclusive: val_size means 'carve this fraction/count out of the train set', which is meaningless when eval data is already supplied.

Source

Thrown at src/llamafactory/data/data_utils.py:100

        raise ValueError(f"Unknown mixing strategy: {data_args.mix_strategy}.")


def split_dataset(
    dataset: Optional[Union["Dataset", "IterableDataset"]],
    eval_dataset: Optional[Union["Dataset", "IterableDataset", dict[str, "Dataset"]]],
    data_args: "DataArguments",
    seed: int,
) -> tuple[dict, dict]:
    r"""Split the dataset and returns two dicts containing train set and validation set.

    Support both map dataset and iterable dataset.

    Returns:
        train_dict: Dictionary containing training data with key "train"
        eval_dict: Dictionary containing evaluation data with keys "validation" or "validation_{name}"
    """
    if eval_dataset is not None and data_args.val_size > 1e-6:
        raise ValueError("Cannot specify `val_size` if `eval_dataset` is not None.")

    # the train and eval better to in dict dtype and separately return for cpode clearly and good handle outside
    train_dict, eval_dict = {}, {}

    if dataset is not None:
        if data_args.streaming:
            dataset = dataset.shuffle(buffer_size=data_args.buffer_size, seed=seed)

        if data_args.val_size > 1e-6:
            if data_args.streaming:
                eval_dict["validation"] = dataset.take(int(data_args.val_size))
                train_dict["train"] = dataset.skip(int(data_args.val_size))
            else:
                val_size = int(data_args.val_size) if data_args.val_size > 1 else data_args.val_size
                split_result = dataset.train_test_split(test_size=val_size, seed=seed)
                train_dict["train"] = split_result["train"]
                eval_dict["validation"] = split_result["test"]
        else:

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove val_size (or set val_size: 0) from the training YAML if you supply an eval dataset.
2. Conversely, drop the eval dataset if you want val_size-based splitting.
3. Never set both — the parser will not fix this for you.

Example fix

# before (yaml)
dataset: train_data
eval_dataset: eval_data
val_size: 0.1

# after (yaml)
dataset: train_data
eval_dataset: eval_data

if eval_dataset is not None:
    assert data_args.val_size <= 1e-6, "set val_size: 0 when eval_dataset is given"

When it happens

Trigger: A training config/run that passes eval_dataset (e.g. a separate eval dataset from dataset_info) together with val_size > 0 in the YAML — common when users leave val_size set after switching from splitting to an explicit eval set.

Common situations: Reusing a base YAML that had val_size: 0.1 and then adding a dedicated eval dataset; tutorial configs carried forward while the data setup changed.

Related errors

• Unsupported model type: {getattr(config, 'model_type')}.
• Unknown mixing strategy: {data_args.mix_strategy}.
• Template is required for MultiModalDataCollator.
• Dataset converter {name} not found.
• Unsupported protocol in path: {path}. Use 's3://' or 'gs://'

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a609b109245c0a7e. Report an issue: GitHub.