sgl-project/sglang · critical · RuntimeError

PD peers must connect matching DCP ranks, got prefill={self.

Error message

PD peers must connect matching DCP ranks, got prefill={self.dcp_rank}, decode={dst_dcp_rank}

What it means

In DCP (decode-context-parallel) disaggregation with equal prefill and decode DCP sizes, ranks must pair 1:1 (prefill rank i transfers to decode rank i). requires_dcp_relayout raises RuntimeError when a bootstrap peer reports a different dcp_rank with the same dcp_size, which would mean miswired peer discovery.

Source

Thrown at python/sglang/srt/disaggregation/common/conn.py:312

    def _should_skip_cp_replicated_state_transfer(self) -> bool:
        """Whether this prefill rank should omit CP-replicated state.

        Prefill CP materializes global token order before writing state pools, so
        every CP rank holds the same state. When all CP ranks transfer their KV
        shards, only rank 0 needs to send that state. Cache layer split is the
        exception because each CP rank owns different state layers.
        """
        return (
            self.attn_cp_size > 1
            and self.attn_cp_rank != 0
            and not get_parallel().enable_dsa_cache_layer_split
        )

    def requires_dcp_relayout(self, dst_dcp_size: int, dst_dcp_rank: int) -> bool:
        if self.dcp_size == dst_dcp_size:
            if self.dcp_rank != dst_dcp_rank:
                raise RuntimeError(
                    "PD peers must connect matching DCP ranks, got "
                    f"prefill={self.dcp_rank}, decode={dst_dcp_rank}"
                )
            return False

        if (
            self.dcp_size == 1
            and dst_dcp_size > 1
            and (self.is_mla_backend or self.is_hybrid_mla_backend)
        ):
            return True

        raise RuntimeError(
            f"Unsupported PD DCP topology: {self.dcp_size} -> {dst_dcp_size}"
        )

    def prepare_dcp_token_item_lens(self, dst_page_item_lens: List[int]) -> List[int]:
        page_size = self.kv_args.page_size

View on GitHub (pinned to 0132848349)

Solutions

1. Clear stale bootstrap metadata (etcd/redis keys) and restart both PD clusters so ranks re-register cleanly
2. Verify both sides run identical --dcp-size and that rank assignment (e.g. CP/DCP rank derivation) is deterministic and matches
3. Ensure only one prefill cluster and one decode cluster share the bootstrap namespace; isolate environments otherwise

if dcp_size == dst_dcp_size:\n    assert dcp_rank == dst_dcp_rank, f'rank pairing mismatch {dcp_rank} vs {dst_dcp_rank}; clear bootstrap metadata'

try:\n    conn.requires_dcp_relayout(dst_dcp_size, dst_dcp_rank)\nexcept RuntimeError:\n    # re-bootstrap: purge stale peer entries and re-register this rank\n    metadata_store.clear_pd_entries(); rebootstrap()

When it happens

Trigger: During PD bootstrap (_add_remote_peer / bootstrap_thread) with dcp_size == dst_dcp_size but self.dcp_rank != dst_dcp_rank — e.g. prefill DCP rank 0 bootstrapping against decode DCP rank 1.

Common situations: Bootstrap/metadata store has stale entries from a previous cluster layout so a prefill node pairs with the wrong decode rank; nodes restarted with different --dcp-size or rank assignments while old sessions persist; network address reuse across restarts.

Related errors

• Unsupported PD DCP topology: {self.dcp_size} -> {dst_dcp_siz
• PD DCP source/destination KV geometry differs: src={src_toke
• PD decode DCP requires an MLA or hybrid-MLA KV pool.
• PD decode DCP currently requires prefill attention CP=1, got
• PD DCP transfer requires decode_prefix_len to align to the v

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c56dcf555fd34341. Report an issue: GitHub.