{"record":{"id":"05c7f904b8ef4c20","repo":"sgl-project/sglang","slug":"group-group-name-is-destroyed","errorCode":null,"errorMessage":"Group {group_name} is destroyed.","messagePattern":"Group (.+?) is destroyed\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/distributed/parallel_state.py","lineNumber":123,"sourceCode":"            )\n            tensor_list.append(value)\n        else:\n            metadata_list.append((key, value))\n    return metadata_list, tensor_list\n\n\n_groups: dict[str, Callable[[], Optional[\"GroupCoordinator\"]]] = {}\n\n\ndef _register_group(group: \"GroupCoordinator\") -> None:\n    _groups[group.unique_name] = weakref.ref(group)\n\n\ndef all_reduce(tensor: torch.Tensor, group_name: str) -> torch.Tensor:\n    assert group_name in _groups, f\"Group {group_name} is not found.\"\n    group = _groups[group_name]()\n    if group is None:\n        raise ValueError(f\"Group {group_name} is destroyed.\")\n    return group._all_reduce_out_place(tensor)\n\n\ndef all_reduce_fake(tensor: torch.Tensor, group_name: str) -> torch.Tensor:\n    return torch.empty_like(tensor)\n\n\ndef get_world_group() -> GroupCoordinator:\n    assert _WORLD is not None, \"world group is not initialized\"\n    return _WORLD\n\n\ndef world_group_is_initialized() -> bool:\n    return _WORLD is not None\n\n\ndef init_world_group(\n    ranks: list[int], local_rank: int, backend: str","sourceCodeStart":105,"sourceCodeEnd":141,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/distributed/parallel_state.py#L105-L141","documentation":"Raised by all_reduce() when the named process group exists in the registry but its stored weak reference has been garbage collected, meaning the underlying torch.distributed group was destroyed. The library keeps groups in a _groups dict mapping name -> weakref, so a destroyed/collected group yields None. It signals the collective is being issued against a group whose lifetime has ended.","triggerScenarios":"Calling all_reduce(tensor, group_name) after the group object was destroyed or garbage collected, e.g. after destroy_model_parallel / teardown, or from a worker process re-using a stale group name after re-initialization.","commonSituations":"Re-initializing distributed state (restart, failover, test reruns in the same process) while holding references to old group names; CUDA/distributed teardown racing with an in-flight all_reduce; calling is_the_same_node_as after the group was freed.","solutions":["Ensure the distributed environment and model parallel groups are (re-)initialized before issuing collectives (call the init routine that populates _groups)","Check group liveness before use, or re-create the destroyed group, then retry the all_reduce","Avoid caching group names across teardown/reinit cycles; re-acquire them after destroy_model_parallel()"],"exampleFix":"// before\nimport parallel_state as ps\nout = ps.all_reduce(t, \"tp\")  # after groups were destroyed\n\n// after\nps.initialize_model_parallel(...)  # re-create groups first\nout = ps.all_reduce(t, \"tp\")","handlingStrategy":"validation","validationCode":"import parallel_state as ps\nfrom ps import _groups  # or an is_group_alive helper if exposed\ngroup = _groups.get(group_name)\nif group_name not in _groups or group() is None:\n    raise RuntimeError(f\"group '{group_name}' unavailable; re-initialize distributed state\")\nout = ps.all_reduce(t, group_name)","typeGuard":null,"tryCatchPattern":"try:\n    out = ps.all_reduce(t, group_name)\nexcept ValueError as e:\n    if \"is destroyed\" in str(e):\n        ps.initialize_model_parallel(...)  # re-init then retry once\n    else:\n        raise","preventionTips":["Re-initialize distributed groups after any teardown/destroy call","Never cache group names across process restarts or re-init cycles","Tear down and recreate groups atomically; quiesce in-flight collectives first"],"tags":["distributed","collective","process-group","weakref"],"backgroundTag":"distributed-process-group-destroyed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}