{"record":{"id":"0ce232aee4948279","repo":"hiyouga/LlamaFactory","slug":"ppo-training-is-incompatible-with-s-2-attn","errorCode":null,"errorMessage":"PPO training is incompatible with S^2-Attn.","messagePattern":"PPO training is incompatible with S\\^2-Attn\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":455,"sourceCode":"            raise ValueError(\"Megatron Bridge only supports `full` and `lora` finetuning.\")\n        if model_args.quantization_bit is not None:\n            raise ValueError(\"Quantized models are not supported with Megatron Bridge.\")\n        if training_args.deepspeed is not None:\n            raise ValueError(\"Megatron Bridge is incompatible with DeepSpeed.\")\n        if mb_args is None:\n            raise ValueError(\"Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.\")\n        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)\n        finetuning_args.megatron_bridge_args = mb_args\n\n    if finetuning_args.stage in [\"rm\", \"ppo\"] and training_args.load_best_model_at_end:\n        raise ValueError(\"RM and PPO stages do not support `load_best_model_at_end`.\")\n\n    if finetuning_args.stage == \"ppo\":\n        if not training_args.do_train:\n            raise ValueError(\"PPO training does not support evaluation, use the SFT stage to evaluate models.\")\n\n        if model_args.shift_attn:\n            raise ValueError(\"PPO training is incompatible with S^2-Attn.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_kt:\n            raise ValueError(\"KTransformers does not support lora reward model.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_unsloth:\n            raise ValueError(\"Unsloth does not support lora reward model.\")\n\n        if training_args.report_to and any(\n            logger not in (\"wandb\", \"tensorboard\", \"trackio\", \"none\") for logger in training_args.report_to\n        ):\n            raise ValueError(\"PPO only accepts wandb, tensorboard, or trackio logger.\")\n\n    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:\n        raise ValueError(\"Please launch distributed training with `llamafactory-cli` or `torchrun`.\")\n\n    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:\n        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n","sourceCodeStart":437,"sourceCodeEnd":473,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L437-L473","documentation":"shift_attn enables S^2-Attn (shifted attention, a training trick for long context without full flash attention support) by monkey-patching attention layers. The PPO training loop's rollout/critic structure is incompatible with that patch, so stage ppo + model_args.shift_attn is rejected.","triggerScenarios":"stage: ppo together with shift_attn: true in model_args of the training config, parsed by get_train_args().","commonSituations":"Long-context SFT recipes that enable shift_attn being repurposed for PPO; users combining every memory/length optimization flag at once.","solutions":["Set shift_attn: false for PPO runs.","For long-context PPO, rely on flash_attention_2 (flash_attn: fa2) and cutoff_len instead.","Keep shift_attn only in sft/pt long-context configs."],"exampleFix":"# before\nstage: ppo\nshift_attn: true\n\n# after\nstage: ppo\nshift_attn: false\nflash_attn: fa2","handlingStrategy":"validation","validationCode":"if cfg.get(\"stage\") == \"ppo\" and cfg.get(\"shift_attn\"):\n    raise SystemExit(\"shift_attn (S^2-Attn) is incompatible with PPO; use flash_attn: fa2 instead\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Confine long-context trick flags (shift_attn, enable_liger_kernel variants) to sft/pt configs.","Maintain a stage->forbidden-flags table in your config validator."],"tags":["config","ppo","attention","long-context","validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}