{"record":{"id":"ae45fc6d43bf227c","repo":"hiyouga/LlamaFactory","slug":"dpo-label-smoothing-is-only-valid-for-sigmoid-lo","errorCode":null,"errorMessage":"`dpo_label_smoothing` is only valid for sigmoid loss function.","messagePattern":"`dpo_label_smoothing` is only valid for sigmoid loss function\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/finetuning_args.py","lineNumber":610,"sourceCode":"        self.apollo_target: list[str] = split_arg(self.apollo_target)\n        self.use_ref_model = self.stage == \"dpo\" and self.pref_loss not in [\"orpo\", \"simpo\"]\n\n        assert self.finetuning_type in [\"lora\", \"oft\", \"freeze\", \"full\"], \"Invalid fine-tuning method.\"\n        assert self.ref_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.reward_model_quantization_bit in [None, 8, 4], \"We only accept 4-bit or 8-bit quantization.\"\n        assert self.hyper_parallel_cp_size > 0, \"`hyper_parallel_cp_size` must be greater than 0.\"\n\n        if self.stage == \"ppo\" and self.reward_model is None:\n            raise ValueError(\"`reward_model` is necessary for PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"lora\" and self.finetuning_type != \"lora\":\n            raise ValueError(\"`reward_model_type` cannot be lora for Freeze/Full PPO training.\")\n\n        if self.stage == \"ppo\" and self.reward_model_type == \"oft\" and self.finetuning_type != \"oft\":\n            raise ValueError(\"`reward_model_type` cannot be oft for Freeze/Full PPO training.\")\n\n        if self.stage == \"dpo\" and self.pref_loss != \"sigmoid\" and self.dpo_label_smoothing > 1e-6:\n            raise ValueError(\"`dpo_label_smoothing` is only valid for sigmoid loss function.\")\n\n        if self.use_llama_pro and self.finetuning_type == \"full\":\n            raise ValueError(\"`use_llama_pro` is only valid for Freeze or LoRA training.\")\n\n        if self.finetuning_type == \"lora\" and (self.use_galore or self.use_apollo or self.use_badam):\n            raise ValueError(\"Cannot use LoRA with GaLore, APOLLO or BAdam together.\")\n\n        if int(self.use_galore) + int(self.use_apollo) + (self.use_badam) > 1:\n            raise ValueError(\"Cannot use GaLore, APOLLO or BAdam together.\")\n\n        if self.pissa_init and (self.stage in [\"ppo\", \"kto\"] or self.use_ref_model):\n            raise ValueError(\"Cannot use PiSSA for current training stage.\")\n\n        if self.finetuning_type != \"lora\":\n            if self.loraplus_lr_ratio is not None:\n                raise ValueError(\"`loraplus_lr_ratio` is only valid for LoRA training.\")\n\n            if self.use_rslora:","sourceCodeStart":592,"sourceCodeEnd":628,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/finetuning_args.py#L592-L628","documentation":"Label smoothing for DPO is only implemented for the original sigmoid (Bradley-Terry) loss. The alternative preference losses (orpo, simpo,IPO-style variants) compute their own calibration and ignore dpo_label_smoothing, so FinetuningArguments.__post_init__ (src/llamafactory/hparams/finetuning_args.py:610) rejects the combination to prevent silently ignored settings.","triggerScenarios":"stage: dpo with pref_loss set to something other than sigmoid (e.g. orpo, simpo, ipo) while dpo_label_smoothing > 0 (any value above the 1e-6 epsilon).","commonSituations":"Tuning pref_loss for better performance but keeping dpo_label_smoothing from an earlier sigmoid run. Also copying hyperparameters from DPO papers/blogs that used label smoothing into an ORPO/SimPO config.","solutions":["Remove dpo_label_smoothing (or set it to 0) if you want to keep the non-sigmoid pref_loss.","Set pref_loss: sigmoid if label smoothing is important to your run.","If you thought smoothing applied to your loss, re-read the chosen loss's paper — ORPO/SimPO have their own regularization."],"exampleFix":"# before (yaml)\nstage: dpo\npref_loss: simpo\ndpo_label_smoothing: 0.1\n\n# after (yaml)\nstage: dpo\npref_loss: simpo\n# dpo_label_smoothing removed","handlingStrategy":"validation","validationCode":"def check_dpo_smoothing(stage: str, pref_loss: str, dpo_label_smoothing: float) -> None:\n    if stage == \"dpo\" and pref_loss != \"sigmoid\" and dpo_label_smoothing > 1e-6:\n        raise ValueError(f\"dpo_label_smoothing is ignored by pref_loss={pref_loss}; remove it\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["When changing pref_loss, sweep the config for sigmoid-specific hyperparameters (label smoothing).","Maintain one canonical YAML per preference-loss variant instead of editing a single file."],"tags":["config","dpo","preference-learning","hyperparameters"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}