hiyouga/LlamaFactory · error · ValueError

dp_size must be divisible by ep_size, got {dp_size} % {ep_si

Error message

dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.

What it means

FSDPTurbo builds an expert mesh of shape (edp, efsdp, ep, cp) from dp_size and ep_size; this requires dp_size to divide evenly by ep_size. If world size is not a multiple of the requested expert-parallel degree, ranks cannot be partitioned and initialization aborts.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:62

        self.expert_mesh: DeviceMesh | None = None
        self.edp_mesh: DeviceMesh | None = None
        self.efsdp_mesh: DeviceMesh | None = None
        self.ep_mesh: DeviceMesh | None = None
        self.expert_cp_mesh: DeviceMesh | None = None

    @property
    def initialized(self) -> bool:
        return self._initialized

    def initialize(self, dist_interface: DistributedInterface, dist_config: dict) -> None:
        dp_size = dist_interface.get_world_size(Dim.DP)
        cp_size = dist_interface.strategy.cp_size
        ep_size = int(dist_config.get("ep_size", 1))

        if ep_size < 1:
            raise ValueError(f"ep_size must be positive, got {ep_size}.")
        if dp_size % ep_size != 0:
            raise ValueError(f"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.")

        topology = (dp_size, cp_size, ep_size)
        if self._initialized:
            current_topology = (self.dp_size, self.cp_size, self.ep_size)
            if topology != current_topology:
                raise RuntimeError(
                    f"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}."
                )
            return

        self.dp_size = dp_size
        self.cp_size = cp_size
        self.ep_size = ep_size

        if ep_size > 1:
            self.efsdp_size = dp_size // ep_size
            self.edp_size = dp_size // (ep_size * self.efsdp_size)
            if dist_interface.get_device_mesh(Dim.DP) is None:

View on GitHub (pinned to f28afaf635)

Solutions

1. Choose ep_size that divides dp_size (from a dp of 8: 1, 2, 4, 8)
2. Adjust world size or other parallel dims so the division is exact
3. Remember dp_size here is the DP world size after other dims, not total GPUs

Example fix

# before (8 GPUs, dp=8)
torchrun --nproc_per_node 8 train.py  # dist_config: {"ep_size": 3}

# after
dist_config = {"ep_size": 4}  # 8 % 4 == 0

dp = dist_interface.get_world_size(Dim.DP)
ep = int(dist_config.get("ep_size", 1))
assert dp % ep == 0, f"dp_size={dp} not divisible by ep_size={ep}; valid ep: {[d for d in range(1, dp + 1) if dp % d == 0]}"

When it happens

Trigger: e.g. world_size=8 with default dp_size=8 and ep_size=3, or any combination where dp_size % ep_size != 0.

Common situations: User sets ep_size to the number of experts or number of nodes rather than a divisor of data-parallel size; changes GPU count (torchrun nproc) without revisiting ep_size; tensor/other parallel dims shrink dp_size unexpectedly.

Related errors

• ep_size must be positive, got {ep_size}.
• FSDPTurbo parallel state is already initialized with {curren
• FSDPTurbo expert parallelism requires an initialized distrib
• FSDPTurbo parallel state must be initialized before clipping
• `tensor_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/744a57006415a97c. Report an issue: GitHub.