{"record":{"id":"6bed73faab0f56fe","repo":"sgl-project/sglang","slug":"torch-distributed-must-be-initialised-before-commo","errorCode":null,"errorMessage":"torch.distributed must be initialised before CommonKVManager registers to the bootstrap server in multi-node prefill mode.","messagePattern":"torch\\.distributed must be initialised before CommonKVManager registers to the bootstrap server in multi-node prefill mode\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/common/conn.py","lineNumber":755,"sourceCode":"        info.target_pp_ranks = target_pp_ranks\n        info.required_dst_info_num = required_dst_info_num\n        info.required_prefill_response_num = required_prefill_response_num\n\n    def _sync_bootstrap_port_across_nodes(self, local_port: int) -> int:\n        \"\"\"Broadcast world-rank-0's bootstrap port to all prefill ranks.\n\n        Required for multi-node prefill when the launcher auto-reserves a\n        free port per host (e.g. Dynamo's\n        `_reserve_disaggregation_bootstrap_port`): without sync, non-leader\n        ranks register to `<leader_ip>:<their_local_port>`, hit\n        `Connection refused`, and the leader's `prefill_port_table` ends\n        up missing rows.\n        \"\"\"\n        if not self.dist_init_addr or get_parallel().nnodes == 1:\n            return local_port\n\n        if not (dist.is_available() and dist.is_initialized()):\n            raise RuntimeError(\n                \"torch.distributed must be initialised before \"\n                \"CommonKVManager registers to the bootstrap server in \"\n                \"multi-node prefill mode.\"\n            )\n\n        world_group = get_world_group()\n        synced_port = world_group.broadcast_object(local_port, src=0)\n        if synced_port != local_port:\n            logger.info(\n                f\"Synced disaggregation bootstrap port from leader: \"\n                f\"local={local_port} -> leader={synced_port} \"\n                f\"(world_rank={world_group.rank_in_group})\"\n            )\n        return synced_port\n\n    def register_to_bootstrap(self):\n        \"\"\"Register prefill server info to bootstrap server via HTTP PUT.\"\"\"\n        if self.dist_init_addr:","sourceCodeStart":737,"sourceCodeEnd":773,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/common/conn.py#L737-L773","documentation":"Thrown by _sync_bootstrap_port_across_nodes when a multi-node prefill CommonKVManager needs to broadcast its bootstrap port across ranks but torch.distributed is not initialized. Multi-node prefill requires the process group to exchange the bootstrap registration port before all ranks can register consistently.","triggerScenarios":"dist_init_addr is set and nnodes > 1 (multi-node prefill) but dist.is_initialized() is False when CommonKVManager.__init__ runs — i.e. the manager is constructed before init_distributed_model / torch.distributed.init_process_group completes.","commonSituations":"Custom serving entrypoints or tests that instantiate the KV manager early; refactors that reordered initialization; a missing/failed torchrun or --dist-init-addr setup so process-group init never happened.","solutions":["Ensure torch.distributed.init_process_group (SGLang's standard distributed init) runs before creating the disaggregation KV manager","Launch multi-node prefill via the supported launcher (torchrun / sglang multi-node scripts) so the PG exists in every rank","If single-node, remove the stray --dist-init-addr / nnodes>1 configuration that triggers the multi-node path"],"exampleFix":"# before\nkv_manager = CommonKVManager(...)  # created before distributed init\ninit_process_group(backend='nccl', ...)\n# after\ninit_process_group(backend='nccl', ...)\nkv_manager = CommonKVManager(...)","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\nif nnodes > 1:\n    assert dist.is_available() and dist.is_initialized(), 'init process group before KV manager'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always run init_distributed before constructing disaggregation managers","Use the standard sglang launcher instead of custom entrypoints for multi-node"],"tags":["disaggregation","multi-node","torch-distributed","initialization-order"],"backgroundTag":"distributed-not-initialized","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T11:17:15.048Z"}