{"record":{"id":"fe70da1ee1463f3e","repo":"hiyouga/LlamaFactory","slug":"rm-and-ppo-stages-do-not-support-load-best-model","errorCode":null,"errorMessage":"RM and PPO stages do not support `load_best_model_at_end`.","messagePattern":"RM and PPO stages do not support `load_best_model_at_end`\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":448,"sourceCode":"\n    if finetuning_args.use_megatron_bridge:\n        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:\n            raise ValueError(\"Megatron Bridge cannot be used together with MCA or HyperParallel.\")\n        if finetuning_args.stage not in [\"pt\", \"sft\"]:\n            raise ValueError(\"Megatron Bridge only supports the `pt` and `sft` stages.\")\n        if finetuning_args.finetuning_type not in [\"full\", \"lora\"]:\n            raise ValueError(\"Megatron Bridge only supports `full` and `lora` finetuning.\")\n        if model_args.quantization_bit is not None:\n            raise ValueError(\"Quantized models are not supported with Megatron Bridge.\")\n        if training_args.deepspeed is not None:\n            raise ValueError(\"Megatron Bridge is incompatible with DeepSpeed.\")\n        if mb_args is None:\n            raise ValueError(\"Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.\")\n        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)\n        finetuning_args.megatron_bridge_args = mb_args\n\n    if finetuning_args.stage in [\"rm\", \"ppo\"] and training_args.load_best_model_at_end:\n        raise ValueError(\"RM and PPO stages do not support `load_best_model_at_end`.\")\n\n    if finetuning_args.stage == \"ppo\":\n        if not training_args.do_train:\n            raise ValueError(\"PPO training does not support evaluation, use the SFT stage to evaluate models.\")\n\n        if model_args.shift_attn:\n            raise ValueError(\"PPO training is incompatible with S^2-Attn.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_kt:\n            raise ValueError(\"KTransformers does not support lora reward model.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_unsloth:\n            raise ValueError(\"Unsloth does not support lora reward model.\")\n\n        if training_args.report_to and any(\n            logger not in (\"wandb\", \"tensorboard\", \"trackio\", \"none\") for logger in training_args.report_to\n        ):\n            raise ValueError(\"PPO only accepts wandb, tensorboard, or trackio logger.\")","sourceCodeStart":430,"sourceCodeEnd":466,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L430-L466","documentation":"load_best_model_at_end is a HF Trainer feature that restores the checkpoint with the best eval metric after training. For reward modeling (rm) and PPO the value model / policy is mutated during training in ways that make checkpoint restoration meaningless, so the parser rejects the flag for those stages.","triggerScenarios":"stage: rm or stage: ppo together with load_best_model_at_end: true in training_args, run via llamafactory-cli train or run_exp().","commonSituations":"Reusing an SFT hyperparameter YAML (which commonly sets load_best_model_at_end + metric_for_best_model for early stopping) for RM or PPO training without clearing the flag.","solutions":["Set load_best_model_at_end: false (and remove metric_for_best_model / greater_is_better) for rm/ppo runs.","Save checkpoints periodically (save_strategy: steps) and pick the best RM/PPO checkpoint by offline evaluation.","Keep the flag only in sft configs."],"exampleFix":"# before\nstage: rm\nload_best_model_at_end: true\nmetric_for_best_model: eval_loss\n\n# after\nstage: rm\nload_best_model_at_end: false","handlingStrategy":"validation","validationCode":"if cfg.get(\"stage\") in (\"rm\", \"ppo\") and cfg.get(\"load_best_model_at_end\"):\n    raise SystemExit(\"load_best_model_at_end is unsupported for rm/ppo; select checkpoints by offline eval\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Split SFT and RM/PPO config templates; only the SFT template carries early-stopping flags.","When reusing an SFT YAML for RM/PPO, run a key-stripper that removes load_best_model_at_end, metric_for_best_model, greater_is_better."],"tags":["config","validation","reward-model","ppo","checkpoints"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}