hiyouga/LlamaFactory · error · ValueError

bf16 and fp16 cannot be both True.

Error message

bf16 and fp16 cannot be both True.

What it means

In the ref-model-free branch of preference loss computation (finetuning_args.use_ref_model == False), LlamaFactory only implements the 'orpo' (odds_ratio_loss) and 'simpo' (simpo_loss) losses (trainer.py:201). Any other pref_loss value (e.g. 'sigmoid' DPO, 'ipo', 'kto') needs reference log-probs and therefore raises NotImplementedError.

Source

Thrown at scripts/megatron_merge.py:48

def convert_mca_to_hf(
    checkpoint_path: str,
    output_path: str = "./output",
    bf16: bool = False,
    fp16: bool = False,
    convert_model_max_length: int | None = None,
):
    """Convert megatron checkpoint to HuggingFace format.

    Args:
        checkpoint_path: Path to the checkpoint to convert
        output_path: Path to save the converted checkpoint
        bf16: Use bfloat16 precision
        fp16: Use float16 precision
        convert_model_max_length: Change the model_max_length in hf config.json
    """
    if bf16 and fp16:
        raise ValueError("bf16 and fp16 cannot be both True.")

    torch_dtype = None
    if bf16:
        torch_dtype = torch.bfloat16
    elif fp16:
        torch_dtype = torch.float16

    convert_checkpoint_to_hf(checkpoint_path, output_path, torch_dtype=torch_dtype)

    if convert_model_max_length is not None:
        config = AutoConfig.from_pretrained(output_path, trust_remote_code=True)
        config.model_max_length = convert_model_max_length
        config.save_pretrained(output_path)


def convert(
    checkpoint_path: str,
    output_path: str = "./output",

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `pref_loss: orpo` or `pref_loss: simpo` if you want reference-free training
2. Or keep your loss (e.g. sigmoid DPO) and enable the reference model (provide ref model args / do not disable use_ref_model) so the dpo_loss branch is taken

Example fix

# before (YAML)
stage: dpo
pref_loss: sigmoid
# ...ref model disabled -> NotImplementedError

# after (option 1)
pref_loss: simpo
# after (option 2)
pref_loss: sigmoid  # and provide/reference the ref model as in examples/train_lora/dpo_llama3.yaml (lora_plus/ref model setup)

REF_FREE_LOSSES = {'orpo', 'simpo'}
assert finetuning_args.use_ref_model or finetuning_args.pref_loss in REF_FREE_LOSSES, (
    f'loss {finetuning_args.pref_loss!r} needs a reference model; use one of {REF_FREE_LOSSES} for ref-free training'
)

def is_ref_free_loss(loss_type: str) -> bool:
    return loss_type in {"orpo", "simpo"}

try:
    run_dpo(train_args)
except NotImplementedError as e:
    if 'Unknown loss type' in str(e):
        raise SystemExit('Set pref_loss to orpo/simpo, or enable a reference model for this loss') from e
    raise

When it happens

Trigger: DPO training YAML with pref_loss: sigmoid (default DPO loss) or ipo while use_ref_model / ref-related flags disable the reference model path; compute_preference_loss dispatches to the no-ref branch and hits the else clause.

Common situations: Switching a working DPO config to reference-free training and forgetting that only ORPO/SimPO are reference-free; copying a simpo example then changing pref_loss back to sigmoid without re-enabling the ref model.

Related errors

• Unknown pref_loss: {self.pref_loss}
• `dpo_label_smoothing` is only valid for sigmoid loss functio
• Cannot use PiSSA for current training stage.
• Please upgrade `transformers` to 4.34.0
• Unable to process key {key}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/92503cf08e03e8a5. Report an issue: GitHub.