{"record":{"id":"e1c6e851eee43535","repo":"hiyouga/LlamaFactory","slug":"sequence-parallelism-requires-flash-attention-ple","errorCode":null,"errorMessage":"Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`.","messagePattern":"Sequence parallelism requires flash attention\\. Please set `flash_attn: flash_attention_2`\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/base_trainer.py","lineNumber":158,"sourceCode":"        self.state = TrainerState(\n            num_training_steps=self.num_training_steps,\n            global_step=self.global_step,\n            epoch=self._resume_epoch,\n        )\n        # Keep callback state aligned with checkpoint-resumed trainer counters.\n        self.state.global_step = self.global_step\n        self.state.epoch = self._resume_epoch\n\n        if self.args.cp_size > 1:\n            # qwen3.5 is not supported because of the different attention implementation, which will be supported in the future.\n            if model.config.model_type == \"qwen3_5\":\n                raise RuntimeError(\n                    \"Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future.\"\n                )\n            from ..plugins.model_plugins.parallelization.sequence_parallel import SequenceParallelModelPlugin\n\n            if model.config._attn_implementation != \"flash_attention_2\":\n                raise ValueError(\n                    \"Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`.\"\n                )\n\n            SequenceParallelModelPlugin(self.args.cp_mode)(model, self.args.cp_size)\n\n    def _create_batch_generator(self) -> None:\n        if (\n            self.args.batching_strategy == BatchingStrategy.PADDING_FREE\n            and getattr(self.model.config, \"_attn_implementation\", None) != \"flash_attention_2\"\n        ):\n            raise ValueError(\"`padding_free` requires `flash_attn: flash_attention_2`.\")\n\n        self.train_batch_generator = BatchGenerator(\n            dataset=self.train_dataset,\n            renderer=self.renderer,\n            micro_batch_size=self.args.micro_batch_size,\n            global_batch_size=self.args.global_batch_size,\n            cutoff_len=self.args.cutoff_len,","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/base_trainer.py#L140-L176","documentation":"When enabling sequence/context parallelism (`cp_size > 1`) in v1 `BaseTrainer`, the model's `config._attn_implementation` must be `flash_attention_2`. The sequence-parallel plugin rewrites attention into a varlen flash kernel, so SDPA/eager attention cannot be substituted.","triggerScenarios":"Training with `cp_size > 1` while `flash_attn` resolves to `auto`/`sdpa`/`eager` — e.g. flash-attn not installed so transformers fell back to sdpa, or the config explicitly set a non-FA2 value.","commonSituations":"Enabling CP on a machine without the flash-attn wheel; `attn_implementation` auto-negotiated to sdpa for a small model; conflicting settings where `flash_attn: auto` did not land on FA2.","solutions":["Set `flash_attn: flash_attention_2` explicitly in the v1 model args","Install a flash-attn build matching your torch/CUDA versions and verify import: `python -c \"import flash_attn\"`","If flash-attn is unavailable on your platform, drop `cp_size` to 1"],"exampleFix":"# before (yaml)\nmodel:\n  flash_attn: auto\ncp_size: 2\n\n# after (yaml)\nmodel:\n  flash_attn: flash_attention_2\ncp_size: 2","handlingStrategy":"validation","validationCode":"def validate_cp_attention(flash_attn: str, cp_size: int) -> None:\n    if cp_size > 1 and flash_attn != \"flash_attention_2\":\n        raise SystemExit(\"cp_size > 1 requires flash_attn: flash_attention_2\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Install flash-attn in every CP image and assert `import flash_attn` at job start","Never use `flash_attn: auto` with CP — set the value explicitly","Log `model.config._attn_implementation` after load to confirm FA2 was honored"],"tags":["v1","context-parallelism","flash-attention","configuration"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}