hiyouga/LlamaFactory · error · ValueError

Sequence parallelism requires flash attention. Please set `f

Error message

Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`.

What it means

When enabling sequence/context parallelism (`cp_size > 1`) in v1 `BaseTrainer`, the model's `config._attn_implementation` must be `flash_attention_2`. The sequence-parallel plugin rewrites attention into a varlen flash kernel, so SDPA/eager attention cannot be substituted.

Source

Thrown at src/llamafactory/v1/core/base_trainer.py:158

        self.state = TrainerState(
            num_training_steps=self.num_training_steps,
            global_step=self.global_step,
            epoch=self._resume_epoch,
        )
        # Keep callback state aligned with checkpoint-resumed trainer counters.
        self.state.global_step = self.global_step
        self.state.epoch = self._resume_epoch

        if self.args.cp_size > 1:
            # qwen3.5 is not supported because of the different attention implementation, which will be supported in the future.
            if model.config.model_type == "qwen3_5":
                raise RuntimeError(
                    "Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future."
                )
            from ..plugins.model_plugins.parallelization.sequence_parallel import SequenceParallelModelPlugin

            if model.config._attn_implementation != "flash_attention_2":
                raise ValueError(
                    "Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`."
                )

            SequenceParallelModelPlugin(self.args.cp_mode)(model, self.args.cp_size)

    def _create_batch_generator(self) -> None:
        if (
            self.args.batching_strategy == BatchingStrategy.PADDING_FREE
            and getattr(self.model.config, "_attn_implementation", None) != "flash_attention_2"
        ):
            raise ValueError("`padding_free` requires `flash_attn: flash_attention_2`.")

        self.train_batch_generator = BatchGenerator(
            dataset=self.train_dataset,
            renderer=self.renderer,
            micro_batch_size=self.args.micro_batch_size,
            global_batch_size=self.args.global_batch_size,
            cutoff_len=self.args.cutoff_len,

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `flash_attn: flash_attention_2` explicitly in the v1 model args
2. Install a flash-attn build matching your torch/CUDA versions and verify import: `python -c "import flash_attn"`
3. If flash-attn is unavailable on your platform, drop `cp_size` to 1

Example fix

# before (yaml)
model:
  flash_attn: auto
cp_size: 2

# after (yaml)
model:
  flash_attn: flash_attention_2
cp_size: 2

def validate_cp_attention(flash_attn: str, cp_size: int) -> None:
    if cp_size > 1 and flash_attn != "flash_attention_2":
        raise SystemExit("cp_size > 1 requires flash_attn: flash_attention_2")

When it happens

Trigger: Training with `cp_size > 1` while `flash_attn` resolves to `auto`/`sdpa`/`eager` — e.g. flash-attn not installed so transformers fell back to sdpa, or the config explicitly set a non-FA2 value.

Common situations: Enabling CP on a machine without the flash-attn wheel; `attn_implementation` auto-negotiated to sdpa for a small model; conflicting settings where `flash_attn: auto` did not land on FA2.

Related errors

• world_size ({helper.get_world_size()}) must be divisible by
• dp_size * cp_size must equal to world_size, got {self.dp_siz
• Context parallelism currently requires `dist_config.name: fs
• `padding_free` requires `flash_attn: flash_attention_2`.
• world_size ({helper.get_world_size()}) must be divisible by

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e1c6e851eee43535. Report an issue: GitHub.