hiyouga/LlamaFactory · error · ValueError

Unknown pref_loss: {self.pref_loss}

Error message

Unknown pref_loss: {self.pref_loss}

What it means

ValueError in DPOTrainer's loss dispatch (dpo_trainer.py:407): the pref_loss string did not match any implemented branch ('dpo', 'orpo', 'simpo' are handled above the raise). The mismatch is caught at the first training step, not at config parse time, so the failure appears after model setup completes.

Source

Thrown at src/llamafactory/v1/trainers/dpo_trainer.py:407

            losses = self._sigmoid_dpo_loss(
                policy_chosen_logps,
                policy_rejected_logps,
                ref_chosen_logps,
                ref_rejected_logps,
            )
            # DPO rewards: beta * (policy_logps - ref_logps)
            chosen_rewards = (self.pref_beta * (policy_chosen_logps - ref_chosen_logps)).detach()
            rejected_rewards = (self.pref_beta * (policy_rejected_logps - ref_rejected_logps)).detach()
        elif self.pref_loss == "orpo":
            losses = self._odds_ratio_loss(chosen_logps_avg, rejected_logps_avg)
            chosen_rewards = (self.pref_beta * chosen_logps_avg).detach()
            rejected_rewards = (self.pref_beta * rejected_logps_avg).detach()
        elif self.pref_loss == "simpo":
            losses = self._simpo_loss(chosen_logps_avg, rejected_logps_avg)
            chosen_rewards = (self.pref_beta * chosen_logps_avg).detach()
            rejected_rewards = (self.pref_beta * rejected_logps_avg).detach()
        else:
            raise ValueError(f"Unknown pref_loss: {self.pref_loss}")

        if self.pref_ftx > 1e-6:
            sft_loss = -chosen_logps_avg
            losses = losses + self.pref_ftx * sft_loss

        # --- Per-step DPO metrics (matches v0 logging) ---
        self._step_metrics = {
            "rewards/chosen": chosen_rewards.mean().item(),
            "rewards/rejected": rejected_rewards.mean().item(),
            "rewards/accuracies": (chosen_rewards > rejected_rewards).float().mean().item(),
            "rewards/margins": (chosen_rewards - rejected_rewards).mean().item(),
            "logps/chosen": policy_chosen_logps.mean().item(),
            "logps/rejected": policy_rejected_logps.mean().item(),
            "logits/chosen": chosen_logits_mean.item(),
            "logits/rejected": rejected_logits_mean.item(),
        }

        return losses.mean()

View on GitHub (pinned to f28afaf635)

Solutions

1. Set pref_loss to one of the implemented values: 'dpo', 'orpo', or 'simpo' (check the elif chain just above the raise for the current list).
2. For KTO or IPO, use the v0 trainer or a different stage; they are not wired into v1 DPOTrainer.
3. Grep the dpo_trainer.py loss dispatch to confirm which values the installed version supports, since the list changes between releases.

Example fix

# before
finetuning_args:
  pref_loss: ipo

# after
finetuning_args:
  pref_loss: dpo

SUPPORTED = {"dpo", "orpo", "simpo"}  # sync with dpo_trainer.py dispatch
assert args.pref_loss in SUPPORTED, f"pref_loss must be one of {SUPPORTED}, got {args.pref_loss}"

def is_supported_pref_loss(name: str) -> bool:
    return name in {"dpo", "orpo", "simpo"}

When it happens

Trigger: Setting pref_loss to an unimplemented or misspelled value such as 'ipo', 'kto', 'DPO' (case-sensitive), or 'dpo_beta' in the finetuning args.

Common situations: Users assume all losses from v0 (ipo, kto) exist in v1; case differences between docs and the string comparison; copy-pasting a loss name from another framework.

Related errors

• bf16 and fp16 cannot be both True.
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `eval_dataset` is not None.
• `dpo_label_smoothing` is only valid for sigmoid loss functio
• Cannot use PiSSA for current training stage.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f5542a92096c8e94. Report an issue: GitHub.