hiyouga/LlamaFactory · error · ValueError

Context parallelism currently requires `dist_config.name: fs

Error message

Context parallelism currently requires `dist_config.name: fsdp2`.

What it means

In v1 `BaseTrainer`, when `dist_config.name` is `deepspeed` and `cp_size > 1`, training is rejected because context parallelism is only implemented on top of the FSDP2 distributed engine (`dist_config.name: fsdp2`). DeepSpeed sharding does not expose the sequence-parallel hooks the CP implementation needs.

Source

Thrown at src/llamafactory/v1/core/base_trainer.py:102

        else:
            self.num_training_steps = self.args.num_train_epochs * len(self.train_batch_generator)

        if self.args.save_epochs is not None:
            steps_per_epoch = len(self.train_batch_generator)
            self.args.save_steps = max(1, int(steps_per_epoch * self.args.save_epochs))

        if self.args.enable_activation_checkpointing:
            self.model.gradient_checkpointing_enable({"use_reentrant": False})
            # Note: under FSDP2 bf16, encoder-tower nn.LayerNorms are made dtype-safe for the
            # checkpoint recompute inside the FSDP2 engine (see fsdp2.py prepare_model), so the
            # tower keeps activation checkpointing too.

        self._deepspeed_engine = None
        dist_name = self.args.dist_config.name if self.args.dist_config is not None else None

        if dist_name == "deepspeed":
            if self.args.cp_size > 1:
                raise ValueError("Context parallelism currently requires `dist_config.name: fsdp2`.")

            from ..plugins.trainer_plugins.distributed.interface import DistributedPlugin

            self._deepspeed_engine = DistributedPlugin("deepspeed").shard_model(
                self.model,
                self.args.dist_config,
                num_micro_batch=self.train_batch_generator.num_micro_batch,
                micro_batch_size=self.args.micro_batch_size,
            )
            self._init_optimizer()
            self._init_lr_scheduler()
            self.model, self.optimizer, self.lr_scheduler = self._deepspeed_engine.prepare(
                self.model, self.optimizer, self.lr_scheduler
            )
        else:
            # fsdp2 / DDP / no dist
            self._shard_model()
            self._init_optimizer()

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch the distributed engine: `dist_config: {name: fsdp2}` and keep `cp_size > 1`
2. Or keep deepspeed but set `cp_size: 1` and use a smaller context / gradient accumulation instead
3. Verify flash-attn is installed, since CP additionally requires `flash_attention_2`

Example fix

# before (yaml)
dist_config:
  name: deepspeed
cp_size: 2

# after (yaml)
dist_config:
  name: fsdp2
cp_size: 2

def validate_cp_dist(dist_name: str | None, cp_size: int) -> None:
    if cp_size > 1 and dist_name != "fsdp2":
        raise SystemExit("cp_size > 1 requires dist_config.name: fsdp2")

When it happens

Trigger: Configuring `cp_size: 2` (or more) together with `dist_config: {name: deepspeed, ...}` in v1 training args.

Common situations: Wanting ZeRO-style sharding plus long-context sequence parallelism in one config; migrating an FSDP2 CP setup to deepspeed without dropping `cp_size`.

Related errors

• world_size ({helper.get_world_size()}) must be divisible by
• dp_size * cp_size must equal to world_size, got {self.dp_siz
• Sequence parallelism requires flash attention. Please set `f
• world_size ({helper.get_world_size()}) must be divisible by
• mp_replicate_size * mp_shard_size must equal to world_size,

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6c58e2e38528177b. Report an issue: GitHub.