{"record":{"id":"f5542a92096c8e94","repo":"hiyouga/LlamaFactory","slug":"unknown-pref-loss-self-pref-loss","errorCode":null,"errorMessage":"Unknown pref_loss: {self.pref_loss}","messagePattern":"Unknown pref_loss: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/trainers/dpo_trainer.py","lineNumber":407,"sourceCode":"            losses = self._sigmoid_dpo_loss(\n                policy_chosen_logps,\n                policy_rejected_logps,\n                ref_chosen_logps,\n                ref_rejected_logps,\n            )\n            # DPO rewards: beta * (policy_logps - ref_logps)\n            chosen_rewards = (self.pref_beta * (policy_chosen_logps - ref_chosen_logps)).detach()\n            rejected_rewards = (self.pref_beta * (policy_rejected_logps - ref_rejected_logps)).detach()\n        elif self.pref_loss == \"orpo\":\n            losses = self._odds_ratio_loss(chosen_logps_avg, rejected_logps_avg)\n            chosen_rewards = (self.pref_beta * chosen_logps_avg).detach()\n            rejected_rewards = (self.pref_beta * rejected_logps_avg).detach()\n        elif self.pref_loss == \"simpo\":\n            losses = self._simpo_loss(chosen_logps_avg, rejected_logps_avg)\n            chosen_rewards = (self.pref_beta * chosen_logps_avg).detach()\n            rejected_rewards = (self.pref_beta * rejected_logps_avg).detach()\n        else:\n            raise ValueError(f\"Unknown pref_loss: {self.pref_loss}\")\n\n        if self.pref_ftx > 1e-6:\n            sft_loss = -chosen_logps_avg\n            losses = losses + self.pref_ftx * sft_loss\n\n        # --- Per-step DPO metrics (matches v0 logging) ---\n        self._step_metrics = {\n            \"rewards/chosen\": chosen_rewards.mean().item(),\n            \"rewards/rejected\": rejected_rewards.mean().item(),\n            \"rewards/accuracies\": (chosen_rewards > rejected_rewards).float().mean().item(),\n            \"rewards/margins\": (chosen_rewards - rejected_rewards).mean().item(),\n            \"logps/chosen\": policy_chosen_logps.mean().item(),\n            \"logps/rejected\": policy_rejected_logps.mean().item(),\n            \"logits/chosen\": chosen_logits_mean.item(),\n            \"logits/rejected\": rejected_logits_mean.item(),\n        }\n\n        return losses.mean()","sourceCodeStart":389,"sourceCodeEnd":425,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/trainers/dpo_trainer.py#L389-L425","documentation":"ValueError in DPOTrainer's loss dispatch (dpo_trainer.py:407): the pref_loss string did not match any implemented branch ('dpo', 'orpo', 'simpo' are handled above the raise). The mismatch is caught at the first training step, not at config parse time, so the failure appears after model setup completes.","triggerScenarios":"Setting pref_loss to an unimplemented or misspelled value such as 'ipo', 'kto', 'DPO' (case-sensitive), or 'dpo_beta' in the finetuning args.","commonSituations":"Users assume all losses from v0 (ipo, kto) exist in v1; case differences between docs and the string comparison; copy-pasting a loss name from another framework.","solutions":["Set pref_loss to one of the implemented values: 'dpo', 'orpo', or 'simpo' (check the elif chain just above the raise for the current list).","For KTO or IPO, use the v0 trainer or a different stage; they are not wired into v1 DPOTrainer.","Grep the dpo_trainer.py loss dispatch to confirm which values the installed version supports, since the list changes between releases."],"exampleFix":"# before\nfinetuning_args:\n  pref_loss: ipo\n\n# after\nfinetuning_args:\n  pref_loss: dpo","handlingStrategy":"validation","validationCode":"SUPPORTED = {\"dpo\", \"orpo\", \"simpo\"}  # sync with dpo_trainer.py dispatch\nassert args.pref_loss in SUPPORTED, f\"pref_loss must be one of {SUPPORTED}, got {args.pref_loss}\"","typeGuard":"def is_supported_pref_loss(name: str) -> bool:\n    return name in {\"dpo\", \"orpo\", \"simpo\"}","tryCatchPattern":null,"preventionTips":["Validate pref_loss right after parsing the config, before model setup.","Keep an allowlist in your config schema synced with the trainer dispatch."],"tags":["dpo","config","loss-function","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}