{"record":{"id":"c647ab286b459bf1","repo":"hiyouga/LlamaFactory","slug":"fsdpturbo-parallel-state-is-already-initialized-wi","errorCode":null,"errorMessage":"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}.","messagePattern":"FSDPTurbo parallel state is already initialized with (.+?), got (.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":68,"sourceCode":"    @property\n    def initialized(self) -> bool:\n        return self._initialized\n\n    def initialize(self, dist_interface: DistributedInterface, dist_config: dict) -> None:\n        dp_size = dist_interface.get_world_size(Dim.DP)\n        cp_size = dist_interface.strategy.cp_size\n        ep_size = int(dist_config.get(\"ep_size\", 1))\n\n        if ep_size < 1:\n            raise ValueError(f\"ep_size must be positive, got {ep_size}.\")\n        if dp_size % ep_size != 0:\n            raise ValueError(f\"dp_size must be divisible by ep_size, got {dp_size} % {ep_size} != 0.\")\n\n        topology = (dp_size, cp_size, ep_size)\n        if self._initialized:\n            current_topology = (self.dp_size, self.cp_size, self.ep_size)\n            if topology != current_topology:\n                raise RuntimeError(\n                    f\"FSDPTurbo parallel state is already initialized with {current_topology}, got {topology}.\"\n                )\n            return\n\n        self.dp_size = dp_size\n        self.cp_size = cp_size\n        self.ep_size = ep_size\n\n        if ep_size > 1:\n            self.efsdp_size = dp_size // ep_size\n            self.edp_size = dp_size // (ep_size * self.efsdp_size)\n            if dist_interface.get_device_mesh(Dim.DP) is None:\n                raise RuntimeError(\"FSDPTurbo expert parallelism requires an initialized distributed device mesh.\")\n\n            self.expert_mesh = init_device_mesh(\n                device_type=dist_interface.current_device.type,\n                mesh_shape=(self.edp_size, self.efsdp_size, self.ep_size, self.cp_size),\n                mesh_dim_names=(self.EDP, self.EFSDP, self.EP, self.EXPERT_CP),","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L50-L86","documentation":"The FSDPTurbo parallel state is a process-wide singleton; initialize() tolerates re-initialization only when the new (dp, cp, ep) topology matches the existing one. A different topology after initialization means meshes and process groups would be stale/inconsistent, so it raises RuntimeError instead of silently reconfiguring.","triggerScenarios":"Calling initialize() a second time with different dp_size/cp_size/ep_size — e.g. re-running setup in the same process after changing dist_config, or a test harness that reinitializes without teardown.","commonSituations":"Notebook/interactive sessions that rebuild trainers; test suites reusing the process; code that retries initialization with adjusted ep_size after a failure.","solutions":["Restart the training process (torchrun relaunch) with the corrected topology instead of re-initializing in-process","If you control the lifecycle, add explicit teardown/reset of the parallel state before re-init, or guard initialization to run once","Check that the same dist_config is used everywhere a second initialize could come from"],"exampleFix":"# before\nstate.initialize(iface, {\"ep_size\": 2})\nstate.initialize(iface, {\"ep_size\": 4})  # RuntimeError\n\n# after\n# relaunch process with ep_size=4, or reset first if a teardown API exists\nstate.initialize(iface, {\"ep_size\": 2})\n# ... end run, exit process; new run uses ep_size=4","handlingStrategy":"validation","validationCode":"state = get_fsdpturbo_parallel_state()\nnew_topology = (dp_size, cp_size, ep_size)\nif state.initialized:\n    assert new_topology == (state.dp_size, state.cp_size, state.ep_size), \\\n        f\"topology change {new_topology} after init; restart the process instead\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat parallel topology as fixed per process; restart to change it","Initialize distributed strategy exactly once per run","Avoid re-running setup cells in notebooks after editing dist_config"],"tags":["fsdpturbo","distributed","singleton","lifecycle"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}