hiyouga/LlamaFactory · error · ValueError

ep_size must be positive, got {ep_size}.

Error message

ep_size must be positive, got {ep_size}.

What it means

FSDPTurbo's parallel-state initialization reads ep_size (expert parallel size) from dist_config, defaulting to 1. Expert parallelism partitions MoE experts across ranks, so ep_size must be a positive integer; values of 0 or negatives make the mesh shape invalid and are rejected immediately.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:60

        self.efsdp_size = 1
        self.edp_size = 1
        self.expert_mesh: DeviceMesh | None = None
        self.edp_mesh: DeviceMesh | None = None
        self.efsdp_mesh: DeviceMesh | None = None
        self.ep_mesh: DeviceMesh | None = None
        self.expert_cp_mesh: DeviceMesh | None = None

    @property
    def initialized(self) -> bool:
        return self._initialized

    def initialize(self, dist_interface: DistributedInterface, dist_config: dict) -> None:
        dp_size = dist_interface.get_world_size(Dim.DP)
        cp_size = dist_interface.strategy.cp_size
        ep_size = int(dist_config.get("ep_size", 1))

        if ep_size < 1:
            raise ValueError(f"ep_size must be positive, got {ep_size}.")
        if dp_size % ep_size != 0:
            raise ValueError(f"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.")

        topology = (dp_size, cp_size, ep_size)
        if self._initialized:
            current_topology = (self.dp_size, self.cp_size, self.ep_size)
            if topology != current_topology:
                raise RuntimeError(
                    f"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}."
                )
            return

        self.dp_size = dp_size
        self.cp_size = cp_size
        self.ep_size = ep_size

        if ep_size > 1:
            self.efsdp_size = dp_size // ep_size

View on GitHub (pinned to f28afaf635)

Solutions

1. Set ep_size to a positive divisor of dp_size (e.g. 2, 4), or remove the key / set 1 for no expert parallelism
2. Check the resolved value (int(dist_config.get('ep_size', 1))) in a dry run

Example fix

# before
dist_config = {"ep_size": 0}

# after
dist_config = {"ep_size": 1}  # or omit the key

ep = int(dist_config.get("ep_size", 1))
assert ep >= 1, f"ep_size must be >= 1, got {ep}"

When it happens

Trigger: dist_config['ep_size'] set to 0 or a negative number (or a string/malformed value that int() maps to <= 0).

Common situations: User disables EP by setting ep_size: 0 instead of omitting it or using 1; YAML math/env interpolation produces 0; copy-paste from a template with a placeholder value.

Related errors

• dp_size must be divisible by ep_size, got {dp_size} % {ep_si
• DeepSpeed config_file is required in dist_config
• FSDPTurbo parallel state is already initialized with {curren
• FSDPTurbo expert parallelism requires an initialized distrib
• FSDPTurbo parallel state must be initialized before clipping

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/40b1c0a9bbd5ed07. Report an issue: GitHub.