hiyouga/LlamaFactory · error · RuntimeError

FSDPTurbo parallel state must be initialized before clipping

Error message

FSDPTurbo parallel state must be initialized before clipping gradients.

What it means

clip_grad_norm_ for FSDPTurbo is CP/EP-aware: it reduces gradient norms across dp, cp, ep, efsdp and expert-cp process groups taken from the initialized parallel state. Without initialization those groups do not exist, so the function refuses to run rather than computing an incorrect (unreduced) norm. It is a programming-order error, not a config value problem.

Source

Thrown at src/llamafactory/v1/plugins/trainer_plugins/distributed/fsdpturbo.py:141

    for group in groups:
        if group is not None:
            dist.all_reduce(value, op=dist.ReduceOp.SUM, group=group)
    return value


def clip_grad_norm_(model: HFModel, max_norm: float, **kwargs) -> float:
    """CP-aware grad norm clipping for FSDPTurbo EP + EFSDP + outer FSDP2.

    Avoids torch.nn.utils.get_total_norm() since mixed DTensor meshes
    (`dp` vs `efsdp`/`ep`) may hit DTensor stack propagation failures.
    """
    from torch.distributed._tensor import DTensor

    norm_type = float(kwargs.get("norm_type", 2.0))
    dist_interface = DistributedInterface()
    parallel_state = get_fsdpturbo_parallel_state()
    if not parallel_state.initialized:
        raise RuntimeError("FSDPTurbo parallel state must be initialized before clipping gradients.")

    device = dist_interface.current_device
    dp_group = dist_interface.get_group(Dim.DP)
    cp_group = dist_interface.get_group(Dim.CP) if dist_interface.strategy.cp_size > 1 else None
    ep_group = parallel_state.ep_mesh.get_group() if parallel_state.ep_mesh is not None else None
    efsdp_group = parallel_state.efsdp_mesh.get_group() if parallel_state.efsdp_mesh is not None else None
    expert_cp_group = (
        parallel_state.expert_cp_mesh.get_group()
        if parallel_state.expert_cp_mesh is not None and parallel_state.cp_size > 1
        else None
    )

    ep_params: list[torch.nn.Parameter] = []
    non_ep_params: list[torch.nn.Parameter] = []
    for param in model.parameters():
        grad = getattr(param, "grad", None)
        if grad is None:
            continue

View on GitHub (pinned to f28afaf635)

Solutions

1. Ensure the FSDPTurbo backend is initialized (trainer setup / initialize()) before the first backward+clip step
2. In tests, initialize a single-rank topology (ep_size=1, dp=1 with process group) before calling clip_grad_norm_
3. If you did not intend FSDPTurbo clipping, make sure the plain FSDP/torch clip path is used instead

Example fix

# before
loss.backward()
fsdpturbo.clip_grad_norm_(model, 1.0)  # RuntimeError
state.initialize(iface, dist_config)

# after
state.initialize(iface, dist_config)
loss.backward()
fsdpturbo.clip_grad_norm_(model, 1.0)

assert get_fsdpturbo_parallel_state().initialized, \
    "initialize FSDPTurbo parallel state before calling clip_grad_norm_"

When it happens

Trigger: Calling fsdpturbo clip_grad_norm_(model, max_norm) before parallel_state.initialize() has run in this process (parallel_state.initialized is False).

Common situations: Custom training loops that clip gradients before setting up distributed strategy; unit tests exercising the clip function in isolation; reordered trainer hooks where clipping fires before distributed init.

Related errors

• ep_size must be positive, got {ep_size}.
• dp_size must be divisible by ep_size, got {dp_size} % {ep_si
• FSDPTurbo parallel state is already initialized with {curren
• FSDPTurbo expert parallelism requires an initialized distrib
• `tensor_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0e3c5b4d5d5b168e. Report an issue: GitHub.