{"record":{"id":"6c58e2e38528177b","repo":"hiyouga/LlamaFactory","slug":"context-parallelism-currently-requires-dist-confi","errorCode":null,"errorMessage":"Context parallelism currently requires `dist_config.name: fsdp2`.","messagePattern":"Context parallelism currently requires `dist_config\\.name: fsdp2`\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/base_trainer.py","lineNumber":102,"sourceCode":"        else:\n            self.num_training_steps = self.args.num_train_epochs * len(self.train_batch_generator)\n\n        if self.args.save_epochs is not None:\n            steps_per_epoch = len(self.train_batch_generator)\n            self.args.save_steps = max(1, int(steps_per_epoch * self.args.save_epochs))\n\n        if self.args.enable_activation_checkpointing:\n            self.model.gradient_checkpointing_enable({\"use_reentrant\": False})\n            # Note: under FSDP2 bf16, encoder-tower nn.LayerNorms are made dtype-safe for the\n            # checkpoint recompute inside the FSDP2 engine (see fsdp2.py prepare_model), so the\n            # tower keeps activation checkpointing too.\n\n        self._deepspeed_engine = None\n        dist_name = self.args.dist_config.name if self.args.dist_config is not None else None\n\n        if dist_name == \"deepspeed\":\n            if self.args.cp_size > 1:\n                raise ValueError(\"Context parallelism currently requires `dist_config.name: fsdp2`.\")\n\n            from ..plugins.trainer_plugins.distributed.interface import DistributedPlugin\n\n            self._deepspeed_engine = DistributedPlugin(\"deepspeed\").shard_model(\n                self.model,\n                self.args.dist_config,\n                num_micro_batch=self.train_batch_generator.num_micro_batch,\n                micro_batch_size=self.args.micro_batch_size,\n            )\n            self._init_optimizer()\n            self._init_lr_scheduler()\n            self.model, self.optimizer, self.lr_scheduler = self._deepspeed_engine.prepare(\n                self.model, self.optimizer, self.lr_scheduler\n            )\n        else:\n            # fsdp2 / DDP / no dist\n            self._shard_model()\n            self._init_optimizer()","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/base_trainer.py#L84-L120","documentation":"In v1 `BaseTrainer`, when `dist_config.name` is `deepspeed` and `cp_size > 1`, training is rejected because context parallelism is only implemented on top of the FSDP2 distributed engine (`dist_config.name: fsdp2`). DeepSpeed sharding does not expose the sequence-parallel hooks the CP implementation needs.","triggerScenarios":"Configuring `cp_size: 2` (or more) together with `dist_config: {name: deepspeed, ...}` in v1 training args.","commonSituations":"Wanting ZeRO-style sharding plus long-context sequence parallelism in one config; migrating an FSDP2 CP setup to deepspeed without dropping `cp_size`.","solutions":["Switch the distributed engine: `dist_config: {name: fsdp2}` and keep `cp_size > 1`","Or keep deepspeed but set `cp_size: 1` and use a smaller context / gradient accumulation instead","Verify flash-attn is installed, since CP additionally requires `flash_attention_2`"],"exampleFix":"# before (yaml)\ndist_config:\n  name: deepspeed\ncp_size: 2\n\n# after (yaml)\ndist_config:\n  name: fsdp2\ncp_size: 2","handlingStrategy":"validation","validationCode":"def validate_cp_dist(dist_name: str | None, cp_size: int) -> None:\n    if cp_size > 1 and dist_name != \"fsdp2\":\n        raise SystemExit(\"cp_size > 1 requires dist_config.name: fsdp2\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pair cp_size > 1 with dist_config fsdp2 in templates only","Lint configs: cp_size>1 implies fsdp2 and flash_attention_2","Decide up front whether ZeRO (deepspeed) or CP (fsdp2) is the memory strategy"],"tags":["v1","context-parallelism","deepspeed","fsdp2","configuration"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}