{"record":{"id":"0e3c5b4d5d5b168e","repo":"hiyouga/LlamaFactory","slug":"fsdpturbo-parallel-state-must-be-initialized-befor","errorCode":null,"errorMessage":"FSDPTurbo parallel state must be initialized before clipping gradients.","messagePattern":"FSDPTurbo parallel state must be initialized before clipping gradients\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py","lineNumber":141,"sourceCode":"    for group in groups:\n        if group is not None:\n            dist.all_reduce(value, op=dist.ReduceOp.SUM, group=group)\n    return value\n\n\ndef clip_grad_norm_(model: HFModel, max_norm: float, **kwargs) -> float:\n    \"\"\"CP-aware grad norm clipping for FSDPTurbo EP + EFSDP + outer FSDP2.\n\n    Avoids torch.nn.utils.get_total_norm() since mixed DTensor meshes\n    (`dp` vs `efsdp`/`ep`) may hit DTensor stack propagation failures.\n    \"\"\"\n    from torch.distributed._tensor import DTensor\n\n    norm_type = float(kwargs.get(\"norm_type\", 2.0))\n    dist_interface = DistributedInterface()\n    parallel_state = get_fsdpturbo_parallel_state()\n    if not parallel_state.initialized:\n        raise RuntimeError(\"FSDPTurbo parallel state must be initialized before clipping gradients.\")\n\n    device = dist_interface.current_device\n    dp_group = dist_interface.get_group(Dim.DP)\n    cp_group = dist_interface.get_group(Dim.CP) if dist_interface.strategy.cp_size > 1 else None\n    ep_group = parallel_state.ep_mesh.get_group() if parallel_state.ep_mesh is not None else None\n    efsdp_group = parallel_state.efsdp_mesh.get_group() if parallel_state.efsdp_mesh is not None else None\n    expert_cp_group = (\n        parallel_state.expert_cp_mesh.get_group()\n        if parallel_state.expert_cp_mesh is not None and parallel_state.cp_size > 1\n        else None\n    )\n\n    ep_params: list[torch.nn.Parameter] = []\n    non_ep_params: list[torch.nn.Parameter] = []\n    for param in model.parameters():\n        grad = getattr(param, \"grad\", None)\n        if grad is None:\n            continue","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py#L123-L159","documentation":"clip_grad_norm_ for FSDPTurbo is CP/EP-aware: it reduces gradient norms across dp, cp, ep, efsdp and expert-cp process groups taken from the initialized parallel state. Without initialization those groups do not exist, so the function refuses to run rather than computing an incorrect (unreduced) norm. It is a programming-order error, not a config value problem.","triggerScenarios":"Calling fsdpturbo clip_grad_norm_(model, max_norm) before parallel_state.initialize() has run in this process (parallel_state.initialized is False).","commonSituations":"Custom training loops that clip gradients before setting up distributed strategy; unit tests exercising the clip function in isolation; reordered trainer hooks where clipping fires before distributed init.","solutions":["Ensure the FSDPTurbo backend is initialized (trainer setup / initialize()) before the first backward+clip step","In tests, initialize a single-rank topology (ep_size=1, dp=1 with process group) before calling clip_grad_norm_","If you did not intend FSDPTurbo clipping, make sure the plain FSDP/torch clip path is used instead"],"exampleFix":"# before\nloss.backward()\nfsdpturbo.clip_grad_norm_(model, 1.0)  # RuntimeError\nstate.initialize(iface, dist_config)\n\n# after\nstate.initialize(iface, dist_config)\nloss.backward()\nfsdpturbo.clip_grad_norm_(model, 1.0)","handlingStrategy":"validation","validationCode":"assert get_fsdpturbo_parallel_state().initialized, \\\n    \"initialize FSDPTurbo parallel state before calling clip_grad_norm_\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Order custom loops: distributed init -> model setup -> train step (backward, clip)","Guard clipping calls behind an initialized-state check in custom trainers","In unit tests, initialize a single-rank topology first"],"tags":["fsdpturbo","gradient-clipping","distributed","initialization-order"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}