sgl-project/sglang · critical · ValueError

Group {group_name} is destroyed.

Error message

Group {group_name} is destroyed.

What it means

Raised by all_reduce() when the named process group exists in the registry but its stored weak reference has been garbage collected, meaning the underlying torch.distributed group was destroyed. The library keeps groups in a _groups dict mapping name -> weakref, so a destroyed/collected group yields None. It signals the collective is being issued against a group whose lifetime has ended.

Source

Thrown at python/sglang/multimodal_gen/runtime/distributed/parallel_state.py:123

            )
            tensor_list.append(value)
        else:
            metadata_list.append((key, value))
    return metadata_list, tensor_list


_groups: dict[str, Callable[[], Optional["GroupCoordinator"]]] = {}


def _register_group(group: "GroupCoordinator") -> None:
    _groups[group.unique_name] = weakref.ref(group)


def all_reduce(tensor: torch.Tensor, group_name: str) -> torch.Tensor:
    assert group_name in _groups, f"Group {group_name} is not found."
    group = _groups[group_name]()
    if group is None:
        raise ValueError(f"Group {group_name} is destroyed.")
    return group._all_reduce_out_place(tensor)


def all_reduce_fake(tensor: torch.Tensor, group_name: str) -> torch.Tensor:
    return torch.empty_like(tensor)


def get_world_group() -> GroupCoordinator:
    assert _WORLD is not None, "world group is not initialized"
    return _WORLD


def world_group_is_initialized() -> bool:
    return _WORLD is not None


def init_world_group(
    ranks: list[int], local_rank: int, backend: str

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the distributed environment and model parallel groups are (re-)initialized before issuing collectives (call the init routine that populates _groups)
2. Check group liveness before use, or re-create the destroyed group, then retry the all_reduce
3. Avoid caching group names across teardown/reinit cycles; re-acquire them after destroy_model_parallel()

Example fix

// before
import parallel_state as ps
out = ps.all_reduce(t, "tp")  # after groups were destroyed

// after
ps.initialize_model_parallel(...)  # re-create groups first
out = ps.all_reduce(t, "tp")

import parallel_state as ps
from ps import _groups  # or an is_group_alive helper if exposed
group = _groups.get(group_name)
if group_name not in _groups or group() is None:
    raise RuntimeError(f"group '{group_name}' unavailable; re-initialize distributed state")
out = ps.all_reduce(t, group_name)

try:
    out = ps.all_reduce(t, group_name)
except ValueError as e:
    if "is destroyed" in str(e):
        ps.initialize_model_parallel(...)  # re-init then retry once
    else:
        raise

When it happens

Trigger: Calling all_reduce(tensor, group_name) after the group object was destroyed or garbage collected, e.g. after destroy_model_parallel / teardown, or from a worker process re-using a stale group name after re-initialization.

Common situations: Re-initializing distributed state (restart, failover, test reruns in the same process) while holding references to old group names; CUDA/distributed teardown racing with an in-flight all_reduce; calling is_the_same_node_as after the group was freed.

Related errors

• requires #senders == #recvs but got #senders={len(received_l
• world_size must be positive and divide global_heads
• world_size ({world_size}) is less than tensor_parallel_degre
• {type(self).__name__} does not implement ring KV-chunk atten
• UlyssesAttention's all-to-all spans the combined sequence pa

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/05c7f904b8ef4c20. Report an issue: GitHub.