hiyouga/LlamaFactory · error · RuntimeError

Sequence parallel is not supported for qwen3.5 model due to

Error message

Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future.

What it means

A RuntimeError raised in v1 `BaseTrainer` when sequence/context parallelism (`cp_size > 1`) is enabled and the loaded model's `config.model_type` is `qwen3_5`. Qwen3.5 uses a different attention implementation that the sequence-parallel plugin does not yet patch, so the combination is explicitly blocked pending future support.

Source

Thrown at src/llamafactory/v1/core/base_trainer.py:152

        # Callbacks
        self.callback_handler = CallbackHandler([LoggingCallback()], trainer=self)
        for cb in callbacks or []:
            self.callback_handler.add_callback(cb)

        # Callbacks: TrainerState tracks progress across the full run.
        self.state = TrainerState(
            num_training_steps=self.num_training_steps,
            global_step=self.global_step,
            epoch=self._resume_epoch,
        )
        # Keep callback state aligned with checkpoint-resumed trainer counters.
        self.state.global_step = self.global_step
        self.state.epoch = self._resume_epoch

        if self.args.cp_size > 1:
            # qwen3.5 is not supported because of the different attention implementation, which will be supported in the future.
            if model.config.model_type == "qwen3_5":
                raise RuntimeError(
                    "Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future."
                )
            from ..plugins.model_plugins.parallelization.sequence_parallel import SequenceParallelModelPlugin

            if model.config._attn_implementation != "flash_attention_2":
                raise ValueError(
                    "Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`."
                )

            SequenceParallelModelPlugin(self.args.cp_mode)(model, self.args.cp_size)

    def _create_batch_generator(self) -> None:
        if (
            self.args.batching_strategy == BatchingStrategy.PADDING_FREE
            and getattr(self.model.config, "_attn_implementation", None) != "flash_attention_2"
        ):
            raise ValueError("`padding_free` requires `flash_attn: flash_attention_2`.")

View on GitHub (pinned to f28afaf635)

Solutions

Set `cp_size: 1` for Qwen3.5 and rely on data/model parallelism plus a longer `cutoff_len`
Shard the long context across ranks by using a different supported model family if CP is mandatory
Track LlamaFactory updates for qwen3.5 sequence-parallel support before re-enabling CP

Example fix

# before (yaml)
cp_size: 4      # with a qwen3_5 model -> RuntimeError

# after (yaml)
cp_size: 1
# compensate with larger per-rank memory or ZeRO/FSDP2 sharding

Defensive patterns

Strategy: validation

Validate before calling

def validate_model_cp(model_type: str, cp_size: int) -> None:
    if cp_size > 1 and model_type == "qwen3_5":
        raise SystemExit("qwen3.5 does not support sequence parallel yet; set cp_size: 1")

Type guard

def model_supports_cp(model_type: str) -> bool:
    return model_type != "qwen3_5"

Try / catch

try:
    trainer = MyV1Trainer(...)
except RuntimeError as e:
    if "qwen3.5" in str(e):
        args.cp_size = 1  # degrade gracefully to non-CP
        trainer = MyV1Trainer(...)
    else:
        raise

Prevention

Read the model-support matrix before enabling CP on new model families
Gate CP behind a config flag per model type in your launch scripts
Watch LlamaFactory release notes for qwen3.5 SP support

When it happens

Trigger: Training a Qwen3.5-family checkpoint in v1 with `cp_size > 1` (context/sequence parallelism enabled).

Common situations: Long-context fine-tuning of a newly released Qwen3.5 model with a config copied from a Qwen2.5 CP run.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/3219d1ae5724e202. Report an issue: GitHub.