hiyouga/LlamaFactory · error · ValueError

`interleave_probs` is only valid for interleaved mixing.

Error message

`interleave_probs` is only valid for interleaved mixing.

What it means

interleave_probs assigns sampling probabilities per dataset and only makes sense when datasets are mixed with an interleaved strategy (HF datasets interleave_datasets). With mix_strategy: concat the datasets are simply concatenated, so probabilities are meaningless and DataArguments.post_init raises ValueError.

Source

Thrown at src/llamafactory/hparams/data_args.py:167

            if isinstance(arg, str):
                return [item.strip() for item in arg.split(",")]
            return arg

        self.dataset = split_arg(self.dataset)
        self.eval_dataset = split_arg(self.eval_dataset)

        if self.media_dir is None:
            self.media_dir = self.dataset_dir

        if self.dataset is None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `dataset` is None.")

        if self.eval_dataset is not None and self.val_size > 1e-6:
            raise ValueError("Cannot specify `val_size` if `eval_dataset` is not None.")

        if self.interleave_probs is not None:
            if self.mix_strategy == "concat":
                raise ValueError("`interleave_probs` is only valid for interleaved mixing.")

            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))
            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):
                raise ValueError("The length of dataset and interleave probs should be identical.")

            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):
                raise ValueError("The length of eval dataset and interleave probs should be identical.")

        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:
            raise ValueError("Streaming mode should have an integer val size.")

        if self.streaming and self.max_samples is not None:
            raise ValueError("`max_samples` is incompatible with `streaming`.")

        if self.mask_history and self.train_on_prompt:
            raise ValueError("`mask_history` is incompatible with `train_on_prompt`.")

        if self.neat_packing:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set mix_strategy: interleave (or interleave_under, interleave_over) in the same config as interleave_probs.
2. Or remove interleave_probs if you want plain concatenation with equal treatment.
3. Double-check dataset_info.json and the YAML both agree on the mixing strategy.

Example fix

# before
mix_strategy: concat
interleave_probs: 0.7,0.3

# after
mix_strategy: interleave
interleave_probs: 0.7,0.3

if data_args.interleave_probs is not None:
    assert data_args.mix_strategy != "concat", "set mix_strategy to an interleave variant or drop interleave_probs"

When it happens

Trigger: YAML containing both interleave_probs: 0.5,0.5 and mix_strategy: concat (or omitting mix_strategy, since concat is not the interleaving path — the check fires whenever mix_strategy == 'concat').

Common situations: Copy-pasting interleave settings from an interleaved example into a concat-style config; assuming interleave_probs works as weights for concat mixing.

Related errors

• The length of eval dataset and interleave probs should be id
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `dataset` is None.
• Cannot specify `val_size` if `eval_dataset` is not None.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c522d8c9215f581c. Report an issue: GitHub.