hiyouga/LlamaFactory · error · ValueError

`dpo_label_smoothing` is only valid for sigmoid loss functio

Error message

`dpo_label_smoothing` is only valid for sigmoid loss function.

What it means

Label smoothing for DPO is only implemented for the original sigmoid (Bradley-Terry) loss. The alternative preference losses (orpo, simpo,IPO-style variants) compute their own calibration and ignore dpo_label_smoothing, so FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:610) rejects the combination to prevent silently ignored settings.

Source

Thrown at src/llamafactory/hparams/finetuning_args.py:610

        self.apollo_target: list[str] = split_arg(self.apollo_target)
        self.use_ref_model = self.stage == "dpo" and self.pref_loss not in ["orpo", "simpo"]

        assert self.finetuning_type in ["lora", "oft", "freeze", "full"], "Invalid fine-tuning method."
        assert self.ref_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.reward_model_quantization_bit in [None, 8, 4], "We only accept 4-bit or 8-bit quantization."
        assert self.hyper_parallel_cp_size > 0, "`hyper_parallel_cp_size` must be greater than 0."

        if self.stage == "ppo" and self.reward_model is None:
            raise ValueError("`reward_model` is necessary for PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "lora" and self.finetuning_type != "lora":
            raise ValueError("`reward_model_type` cannot be lora for Freeze/Full PPO training.")

        if self.stage == "ppo" and self.reward_model_type == "oft" and self.finetuning_type != "oft":
            raise ValueError("`reward_model_type` cannot be oft for Freeze/Full PPO training.")

        if self.stage == "dpo" and self.pref_loss != "sigmoid" and self.dpo_label_smoothing > 1e-6:
            raise ValueError("`dpo_label_smoothing` is only valid for sigmoid loss function.")

        if self.use_llama_pro and self.finetuning_type == "full":
            raise ValueError("`use_llama_pro` is only valid for Freeze or LoRA training.")

        if self.finetuning_type == "lora" and (self.use_galore or self.use_apollo or self.use_badam):
            raise ValueError("Cannot use LoRA with GaLore, APOLLO or BAdam together.")

        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:
            raise ValueError("Cannot use GaLore, APOLLO or BAdam together.")

        if self.pissa_init and (self.stage in ["ppo", "kto"] or self.use_ref_model):
            raise ValueError("Cannot use PiSSA for current training stage.")

        if self.finetuning_type != "lora":
            if self.loraplus_lr_ratio is not None:
                raise ValueError("`loraplus_lr_ratio` is only valid for LoRA training.")

            if self.use_rslora:

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove dpo_label_smoothing (or set it to 0) if you want to keep the non-sigmoid pref_loss.
2. Set pref_loss: sigmoid if label smoothing is important to your run.
3. If you thought smoothing applied to your loss, re-read the chosen loss's paper — ORPO/SimPO have their own regularization.

Example fix

# before (yaml)
stage: dpo
pref_loss: simpo
dpo_label_smoothing: 0.1

# after (yaml)
stage: dpo
pref_loss: simpo
# dpo_label_smoothing removed

def check_dpo_smoothing(stage: str, pref_loss: str, dpo_label_smoothing: float) -> None:
    if stage == "dpo" and pref_loss != "sigmoid" and dpo_label_smoothing > 1e-6:
        raise ValueError(f"dpo_label_smoothing is ignored by pref_loss={pref_loss}; remove it")

When it happens

Trigger: stage: dpo with pref_loss set to something other than sigmoid (e.g. orpo, simpo, ipo) while dpo_label_smoothing > 0 (any value above the 1e-6 epsilon).

Common situations: Tuning pref_loss for better performance but keeping dpo_label_smoothing from an earlier sigmoid run. Also copying hyperparameters from DPO papers/blogs that used label smoothing into an ORPO/SimPO config.

Related errors

• bf16 and fp16 cannot be both True.
• Cannot use PiSSA for current training stage.
• Unknown pref_loss: {self.pref_loss}
• Please upgrade `transformers` to 4.34.0
• Unable to process key {key}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/ae45fc6d43bf227c. Report an issue: GitHub.