hiyouga/LlamaFactory · error · ValueError

PPO training does not support evaluation, use the SFT stage

Error message

PPO training does not support evaluation, use the SFT stage to evaluate models.

What it means

PPO in LlamaFactory is implemented as an online training loop (policy + reward model + value model); there is no PPO 'evaluation' run type. If do_train is false (e.g. only do_eval/do_predict set), the parser raises this error and points you at the SFT stage for evaluating models.

Source

Thrown at src/llamafactory/hparams/parser.py:452

        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Set do_train: true for PPO runs (that is the only supported mode).
2. To evaluate or chat with a PPO-trained checkpoint, switch stage: sft with the adapter/model path and use chat/api or do_eval there.
3. Remove --do_eval/--do_predict flags from ppo launches.

Example fix

# before
stage: ppo
do_train: false
do_eval: true

# after
stage: ppo
do_train: true
# evaluation of the trained policy: use a separate sft/chat run with the saved adapter

if cfg.get("stage") == "ppo" and not cfg.get("do_train", False):
    raise SystemExit("PPO requires do_train: true; evaluate trained policies via a separate sft/chat run")

When it happens

Trigger: stage: ppo with do_train: false (or absent) and do_eval/do_predict: true in the training args — e.g. invoking a ppo config with --do_eval only.

Common situations: Trying to 'evaluate' a PPO-trained policy by running the ppo stage without training; CI pipelines that append --do_eval --eval_dataset ... to every config uniformly.

Related errors

• `predict_with_generate` cannot be set as True except SFT.
• RM and PPO stages do not support `load_best_model_at_end`.
• PPO training is incompatible with S^2-Attn.
• Invalid role
• Unknown mixing strategy: {data_args.mix_strategy}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e6341e47e680421f. Report an issue: GitHub.