hiyouga/LlamaFactory · error · ValueError

RM and PPO stages do not support `load_best_model_at_end`.

Error message

RM and PPO stages do not support `load_best_model_at_end`.

What it means

load_best_model_at_end is a HF Trainer feature that restores the checkpoint with the best eval metric after training. For reward modeling (rm) and PPO the value model / policy is mutated during training in ways that make checkpoint restoration meaningless, so the parser rejects the flag for those stages.

Source

Thrown at src/llamafactory/hparams/parser.py:448

    if finetuning_args.use_megatron_bridge:
        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:
            raise ValueError("Megatron Bridge cannot be used together with MCA or HyperParallel.")
        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Set load_best_model_at_end: false (and remove metric_for_best_model / greater_is_better) for rm/ppo runs.
2. Save checkpoints periodically (save_strategy: steps) and pick the best RM/PPO checkpoint by offline evaluation.
3. Keep the flag only in sft configs.

Example fix

# before
stage: rm
load_best_model_at_end: true
metric_for_best_model: eval_loss

# after
stage: rm
load_best_model_at_end: false

if cfg.get("stage") in ("rm", "ppo") and cfg.get("load_best_model_at_end"):
    raise SystemExit("load_best_model_at_end is unsupported for rm/ppo; select checkpoints by offline eval")

When it happens

Trigger: stage: rm or stage: ppo together with load_best_model_at_end: true in training_args, run via llamafactory-cli train or run_exp().

Common situations: Reusing an SFT hyperparameter YAML (which commonly sets load_best_model_at_end + metric_for_best_model for early stopping) for RM or PPO training without clearing the flag.

Related errors

• `reward_model` is necessary for PPO training.
• `reward_model_type` cannot be lora for Freeze/Full PPO train
• `reward_model_type` cannot be oft for Freeze/Full PPO traini
• PPO training does not support evaluation, use the SFT stage
• PPO training is incompatible with S^2-Attn.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/fe70da1ee1463f3e. Report an issue: GitHub.