sgl-project/sglang · critical · RuntimeError

torch.distributed must be initialised before CommonKVManager

Error message

torch.distributed must be initialised before CommonKVManager registers to the bootstrap server in multi-node prefill mode.

What it means

Thrown by _sync_bootstrap_port_across_nodes when a multi-node prefill CommonKVManager needs to broadcast its bootstrap port across ranks but torch.distributed is not initialized. Multi-node prefill requires the process group to exchange the bootstrap registration port before all ranks can register consistently.

Source

Thrown at python/sglang/srt/disaggregation/common/conn.py:755

        info.target_pp_ranks = target_pp_ranks
        info.required_dst_info_num = required_dst_info_num
        info.required_prefill_response_num = required_prefill_response_num

    def _sync_bootstrap_port_across_nodes(self, local_port: int) -> int:
        """Broadcast world-rank-0's bootstrap port to all prefill ranks.

        Required for multi-node prefill when the launcher auto-reserves a
        free port per host (e.g. Dynamo's
        `_reserve_disaggregation_bootstrap_port`): without sync, non-leader
        ranks register to `<leader_ip>:<their_local_port>`, hit
        `Connection refused`, and the leader's `prefill_port_table` ends
        up missing rows.
        """
        if not self.dist_init_addr or get_parallel().nnodes == 1:
            return local_port

        if not (dist.is_available() and dist.is_initialized()):
            raise RuntimeError(
                "torch.distributed must be initialised before "
                "CommonKVManager registers to the bootstrap server in "
                "multi-node prefill mode."
            )

        world_group = get_world_group()
        synced_port = world_group.broadcast_object(local_port, src=0)
        if synced_port != local_port:
            logger.info(
                f"Synced disaggregation bootstrap port from leader: "
                f"local={local_port} -> leader={synced_port} "
                f"(world_rank={world_group.rank_in_group})"
            )
        return synced_port

    def register_to_bootstrap(self):
        """Register prefill server info to bootstrap server via HTTP PUT."""
        if self.dist_init_addr:

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure torch.distributed.init_process_group (SGLang's standard distributed init) runs before creating the disaggregation KV manager
2. Launch multi-node prefill via the supported launcher (torchrun / sglang multi-node scripts) so the PG exists in every rank
3. If single-node, remove the stray --dist-init-addr / nnodes>1 configuration that triggers the multi-node path

Example fix

# before
kv_manager = CommonKVManager(...)  # created before distributed init
init_process_group(backend='nccl', ...)
# after
init_process_group(backend='nccl', ...)
kv_manager = CommonKVManager(...)

import torch.distributed as dist
if nnodes > 1:
    assert dist.is_available() and dist.is_initialized(), 'init process group before KV manager'

When it happens

Trigger: dist_init_addr is set and nnodes > 1 (multi-node prefill) but dist.is_initialized() is False when CommonKVManager.__init__ runs — i.e. the manager is constructed before init_distributed_model / torch.distributed.init_process_group completes.

Common situations: Custom serving entrypoints or tests that instantiate the KV manager early; refactors that reordered initialization; a missing/failed torchrun or --dist-init-addr setup so process-group init never happened.

Related errors

• Waiting for main node timeout!
• num_instances must be >= 1, got {num_instances}
• Unknown dispatch policy '{name}'. Available: {list(policies.
• Duplicate request_id: {request_id}
• Unknown request_id: {request_id}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6bed73faab0f56fe. Report an issue: GitHub.