hiyouga/LlamaFactory · error · ValueError

PPO training is incompatible with S^2-Attn.

Error message

PPO training is incompatible with S^2-Attn.

What it means

shift_attn enables S^2-Attn (shifted attention, a training trick for long context without full flash attention support) by monkey-patching attention layers. The PPO training loop's rollout/critic structure is incompatible with that patch, so stage ppo + model_args.shift_attn is rejected.

Source

Thrown at src/llamafactory/hparams/parser.py:455

            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Set shift_attn: false for PPO runs.
2. For long-context PPO, rely on flash_attention_2 (flash_attn: fa2) and cutoff_len instead.
3. Keep shift_attn only in sft/pt long-context configs.

Example fix

# before
stage: ppo
shift_attn: true

# after
stage: ppo
shift_attn: false
flash_attn: fa2

if cfg.get("stage") == "ppo" and cfg.get("shift_attn"):
    raise SystemExit("shift_attn (S^2-Attn) is incompatible with PPO; use flash_attn: fa2 instead")

When it happens

Trigger: stage: ppo together with shift_attn: true in model_args of the training config, parsed by get_train_args().

Common situations: Long-context SFT recipes that enable shift_attn being repurposed for PPO; users combining every memory/length optimization flag at once.

Related errors

• RM and PPO stages do not support `load_best_model_at_end`.
• PPO training does not support evaluation, use the SFT stage
• Invalid role
• Unknown mixing strategy: {data_args.mix_strategy}.
• Cannot specify `val_size` if `eval_dataset` is not None.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0ce232aee4948279. Report an issue: GitHub.