hiyouga/LlamaFactory · error · ValueError

`mask_history` is incompatible with `train_on_prompt`.

Error message

`mask_history` is incompatible with `train_on_prompt`.

What it means

train_on_prompt keeps loss on the prompt tokens, while mask_history trains on only the last response turn by masking earlier dialogue history. Applying both is contradictory (the masked-away history includes prompt tokens), so DataArguments.__post_init__ (src/llamafactory/hparams/data_args.py:183) rejects it. Choose one behavior.

Source

Thrown at src/llamafactory/hparams/data_args.py:183

        if self.interleave_probs is not None:
            if self.mix_strategy == "concat":
                raise ValueError("`interleave_probs` is only valid for interleaved mixing.")

            self.interleave_probs = list(map(float, split_arg(self.interleave_probs)))
            if self.dataset is not None and len(self.dataset) != len(self.interleave_probs):
                raise ValueError("The length of dataset and interleave probs should be identical.")

            if self.eval_dataset is not None and len(self.eval_dataset) != len(self.interleave_probs):
                raise ValueError("The length of eval dataset and interleave probs should be identical.")

        if self.streaming and self.val_size > 1e-6 and self.val_size < 1:
            raise ValueError("Streaming mode should have an integer val size.")

        if self.streaming and self.max_samples is not None:
            raise ValueError("`max_samples` is incompatible with `streaming`.")

        if self.mask_history and self.train_on_prompt:
            raise ValueError("`mask_history` is incompatible with `train_on_prompt`.")

        if self.neat_packing:
            self.packing = True

        if self.packing:
            self.cutoff_len -= 1  # avoid pad_to_multiple_of, needs improve

    def to_dict(self) -> dict[str, Any]:
        return asdict(self)

View on GitHub (pinned to f28afaf635)

Solutions

1. If you want loss only on the latest assistant turn, keep mask_history: true and remove train_on_prompt.
2. If you want loss on the entire sequence including prompts, keep train_on_prompt: true and remove mask_history.
3. Leave both off for the default behavior: loss on all assistant responses in multi-turn data.

Example fix

# before (yaml)
mask_history: true
train_on_prompt: true

# after (yaml)
mask_history: true
# train_on_prompt removed

def check_mask_flags(mask_history: bool, train_on_prompt: bool) -> None:
    if mask_history and train_on_prompt:
        raise ValueError("mask_history and train_on_prompt are mutually exclusive")

When it happens

Trigger: A dataset/config with mask_history: true and train_on_prompt: true simultaneously. Fired during DataArguments validation at startup.

Common situations: Users fine-tuning multi-turn chat models who copy flags from different example configs (one using train_on_prompt for knowledge learning, another using mask_history for last-turn supervision) and end up with both.

Related errors

• The length of packed example should be identical to the cuto
• Streaming mode should have an integer val size.
• `max_samples` is incompatible with `streaming`.
• Can not load dataset from {filepath}.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/60bc613b3ab67c15. Report an issue: GitHub.