hiyouga/LlamaFactory · error · ValueError

The length of eval dataset and interleave probs should be id

Error message

The length of eval dataset and interleave probs should be identical.

What it means

Symmetric to the training-dataset check: when interleave_probs is set and eval_dataset is provided, the number of evaluation datasets must equal the number of probabilities. DataArguments.post_init raises ValueError when len(eval_dataset) != len(interleave_probs).

Source

Thrown at src/llamafactory/hparams/data_args.py:174

        if self.media_dir is None:
            self.media_dir = self.dataset_dir

        if self.dataset is None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `dataset` is None.")

        if self.eval_dataset is not None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `eval_dataset` is not None.")

        if self.interleave_probs is not None:
            if self.mix_strategy == "concat":
                raise ValueError("`interleave_probs` is only valid for interleaved mixing.")

            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))
            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):
                raise ValueError("The length of dataset and interleave probs should be identical.")

            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):
                raise ValueError("The length of eval dataset and interleave probs should be identical.")

        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:
            raise ValueError("Streaming mode should have an integer val size.")

        if self.streaming and self.max_samples is not None:
            raise ValueError("`max_samples` is incompatible with `streaming`.")

        if self.mask_history and self.train_on_prompt:
            raise ValueError("`mask_history` is incompatible with `train_on_prompt`.")

        if self.neat_packing:
            self.packing = True

        if self.packing:
            self.cutoff_len -= 1  # avoid pad_to_multiple_of, needs improve

    def to_dict(self) -> dict[str, Any]:
        return asdict(self)

View on GitHub (pinned to f28afaf635)

Solutions

1. Align counts: either give each eval dataset its own probability by matching lengths, or restructure so train and eval use the same number of interleaved datasets.
2. If eval datasets shouldn't be interleaved, reconsider using interleave_probs at all (it applies to both lists).
3. Validate lengths programmatically before training (see validationCode).

Example fix

# before
dataset: d1,d2,d3
eval_dataset: e1,e2
interleave_probs: 0.5,0.3,0.2

# after
dataset: d1,d2
eval_dataset: e1,e2
interleave_probs: 0.5,0.5

n_eval = len(data_args.eval_dataset) if data_args.eval_dataset else 0
if data_args.interleave_probs is not None and data_args.eval_dataset is not None:
    assert n_eval == len(data_args.interleave_probs), \
        f"{n_eval} eval datasets vs {len(data_args.interleave_probs)} probs"

When it happens

Trigger: Two eval datasets (eval_dataset: e1,e2) with three probabilities matching a three-dataset training list; adding eval datasets without extending/adjusting the shared interleave_probs list (note: one list serves both).

Common situations: Reusing a single interleave_probs for different train/eval dataset counts; trimming eval datasets while keeping training probabilities unchanged.

Related errors

• `interleave_probs` is only valid for interleaved mixing.
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `dataset` is None.
• Cannot specify `val_size` if `eval_dataset` is not None.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f1b5ef29e6422d18. Report an issue: GitHub.