hiyouga/LlamaFactory · error · RuntimeError

FSDPTurbo parallel state is already initialized with {curren

Error message

FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}.

What it means

The FSDPTurbo parallel state is a process-wide singleton; initialize() tolerates re-initialization only when the new (dp, cp, ep) topology matches the existing one. A different topology after initialization means meshes and process groups would be stale/inconsistent, so it raises RuntimeError instead of silently reconfiguring.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:68

    @property
    def initialized(self) -> bool:
        return self._initialized

    def initialize(self, dist_interface: DistributedInterface, dist_config: dict) -> None:
        dp_size = dist_interface.get_world_size(Dim.DP)
        cp_size = dist_interface.strategy.cp_size
        ep_size = int(dist_config.get("ep_size", 1))

        if ep_size < 1:
            raise ValueError(f"ep_size must be positive, got {ep_size}.")
        if dp_size % ep_size != 0:
            raise ValueError(f"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.")

        topology = (dp_size, cp_size, ep_size)
        if self._initialized:
            current_topology = (self.dp_size, self.cp_size, self.ep_size)
            if topology != current_topology:
                raise RuntimeError(
                    f"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}."
                )
            return

        self.dp_size = dp_size
        self.cp_size = cp_size
        self.ep_size = ep_size

        if ep_size > 1:
            self.efsdp_size = dp_size // ep_size
            self.edp_size = dp_size // (ep_size * self.efsdp_size)
            if dist_interface.get_device_mesh(Dim.DP) is None:
                raise RuntimeError("FSDPTurbo expert parallelism requires an initialized distributed device mesh.")

            self.expert_mesh = init_device_mesh(
                device_type=dist_interface.current_device.type,
                mesh_shape=(self.edp_size, self.efsdp_size, self.ep_size, self.cp_size),
                mesh_dim_names=(self.EDP, self.EFSDP, self.EP, self.EXPERT_CP),

View on GitHub (pinned to f28afaf635)

Solutions

1. Restart the training process (torchrun relaunch) with the corrected topology instead of re-initializing in-process
2. If you control the lifecycle, add explicit teardown/reset of the parallel state before re-init, or guard initialization to run once
3. Check that the same dist_config is used everywhere a second initialize could come from

Example fix

# before
state.initialize(iface, {"ep_size": 2})
state.initialize(iface, {"ep_size": 4})  # RuntimeError

# after
# relaunch process with ep_size=4, or reset first if a teardown API exists
state.initialize(iface, {"ep_size": 2})
# ... end run, exit process; new run uses ep_size=4

state = get_fsdpturbo_parallel_state()
new_topology = (dp_size, cp_size, ep_size)
if state.initialized:
    assert new_topology == (state.dp_size, state.cp_size, state.ep_size), \
        f"topology change {new_topology} after init; restart the process instead"

When it happens

Trigger: Calling initialize() a second time with different dp_size/cp_size/ep_size — e.g. re-running setup in the same process after changing dist_config, or a test harness that reinitializes without teardown.

Common situations: Notebook/interactive sessions that rebuild trainers; test suites reusing the process; code that retries initialization with adjusted ep_size after a failure.

Related errors

• ep_size must be positive, got {ep_size}.
• dp_size must be divisible by ep_size, got {dp_size} % {ep_si
• FSDPTurbo parallel state must be initialized before clipping
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c647ab286b459bf1. Report an issue: GitHub.