{"record":{"id":"3219d1ae5724e202","repo":"hiyouga/LlamaFactory","slug":"sequence-parallel-is-not-supported-for-qwen3-5-mod","errorCode":null,"errorMessage":"Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future.","messagePattern":"Sequence parallel is not supported for qwen3\\.5 model due to its different attention implementation, which will be supported in the future\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/base_trainer.py","lineNumber":152,"sourceCode":"        # Callbacks\n        self.callback_handler = CallbackHandler([LoggingCallback()], trainer=self)\n        for cb in callbacks or []:\n            self.callback_handler.add_callback(cb)\n\n        # Callbacks: TrainerState tracks progress across the full run.\n        self.state = TrainerState(\n            num_training_steps=self.num_training_steps,\n            global_step=self.global_step,\n            epoch=self._resume_epoch,\n        )\n        # Keep callback state aligned with checkpoint-resumed trainer counters.\n        self.state.global_step = self.global_step\n        self.state.epoch = self._resume_epoch\n\n        if self.args.cp_size > 1:\n            # qwen3.5 is not supported because of the different attention implementation, which will be supported in the future.\n            if model.config.model_type == \"qwen3_5\":\n                raise RuntimeError(\n                    \"Sequence parallel is not supported for qwen3.5 model due to its different attention implementation, which will be supported in the future.\"\n                )\n            from ..plugins.model_plugins.parallelization.sequence_parallel import SequenceParallelModelPlugin\n\n            if model.config._attn_implementation != \"flash_attention_2\":\n                raise ValueError(\n                    \"Sequence parallelism requires flash attention. Please set `flash_attn: flash_attention_2`.\"\n                )\n\n            SequenceParallelModelPlugin(self.args.cp_mode)(model, self.args.cp_size)\n\n    def _create_batch_generator(self) -> None:\n        if (\n            self.args.batching_strategy == BatchingStrategy.PADDING_FREE\n            and getattr(self.model.config, \"_attn_implementation\", None) != \"flash_attention_2\"\n        ):\n            raise ValueError(\"`padding_free` requires `flash_attn: flash_attention_2`.\")\n","sourceCodeStart":134,"sourceCodeEnd":170,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/base_trainer.py#L134-L170","documentation":"A RuntimeError raised in v1 `BaseTrainer` when sequence/context parallelism (`cp_size > 1`) is enabled and the loaded model's `config.model_type` is `qwen3_5`. Qwen3.5 uses a different attention implementation that the sequence-parallel plugin does not yet patch, so the combination is explicitly blocked pending future support.","triggerScenarios":"Training a Qwen3.5-family checkpoint in v1 with `cp_size > 1` (context/sequence parallelism enabled).","commonSituations":"Long-context fine-tuning of a newly released Qwen3.5 model with a config copied from a Qwen2.5 CP run.","solutions":["Set `cp_size: 1` for Qwen3.5 and rely on data/model parallelism plus a longer `cutoff_len`","Shard the long context across ranks by using a different supported model family if CP is mandatory","Track LlamaFactory updates for qwen3.5 sequence-parallel support before re-enabling CP"],"exampleFix":"# before (yaml)\ncp_size: 4      # with a qwen3_5 model -> RuntimeError\n\n# after (yaml)\ncp_size: 1\n# compensate with larger per-rank memory or ZeRO/FSDP2 sharding","handlingStrategy":"validation","validationCode":"def validate_model_cp(model_type: str, cp_size: int) -> None:\n    if cp_size > 1 and model_type == \"qwen3_5\":\n        raise SystemExit(\"qwen3.5 does not support sequence parallel yet; set cp_size: 1\")","typeGuard":"def model_supports_cp(model_type: str) -> bool:\n    return model_type != \"qwen3_5\"","tryCatchPattern":"try:\n    trainer = MyV1Trainer(...)\nexcept RuntimeError as e:\n    if \"qwen3.5\" in str(e):\n        args.cp_size = 1  # degrade gracefully to non-CP\n        trainer = MyV1Trainer(...)\n    else:\n        raise","preventionTips":["Read the model-support matrix before enabling CP on new model families","Gate CP behind a config flag per model type in your launch scripts","Watch LlamaFactory release notes for qwen3.5 SP support"],"tags":["v1","context-parallelism","qwen3","model-support"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}