{"record":{"id":"c56dcf555fd34341","repo":"sgl-project/sglang","slug":"pd-peers-must-connect-matching-dcp-ranks-got-pref","errorCode":null,"errorMessage":"PD peers must connect matching DCP ranks, got prefill={self.dcp_rank}, decode={dst_dcp_rank}","messagePattern":"PD peers must connect matching DCP ranks, got prefill=(.+?), decode=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/common/conn.py","lineNumber":312,"sourceCode":"\n    def _should_skip_cp_replicated_state_transfer(self) -> bool:\n        \"\"\"Whether this prefill rank should omit CP-replicated state.\n\n        Prefill CP materializes global token order before writing state pools, so\n        every CP rank holds the same state. When all CP ranks transfer their KV\n        shards, only rank 0 needs to send that state. Cache layer split is the\n        exception because each CP rank owns different state layers.\n        \"\"\"\n        return (\n            self.attn_cp_size > 1\n            and self.attn_cp_rank != 0\n            and not get_parallel().enable_dsa_cache_layer_split\n        )\n\n    def requires_dcp_relayout(self, dst_dcp_size: int, dst_dcp_rank: int) -> bool:\n        if self.dcp_size == dst_dcp_size:\n            if self.dcp_rank != dst_dcp_rank:\n                raise RuntimeError(\n                    \"PD peers must connect matching DCP ranks, got \"\n                    f\"prefill={self.dcp_rank}, decode={dst_dcp_rank}\"\n                )\n            return False\n\n        if (\n            self.dcp_size == 1\n            and dst_dcp_size > 1\n            and (self.is_mla_backend or self.is_hybrid_mla_backend)\n        ):\n            return True\n\n        raise RuntimeError(\n            f\"Unsupported PD DCP topology: {self.dcp_size} -> {dst_dcp_size}\"\n        )\n\n    def prepare_dcp_token_item_lens(self, dst_page_item_lens: List[int]) -> List[int]:\n        page_size = self.kv_args.page_size","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/common/conn.py#L294-L330","documentation":"In DCP (decode-context-parallel) disaggregation with equal prefill and decode DCP sizes, ranks must pair 1:1 (prefill rank i transfers to decode rank i). requires_dcp_relayout raises RuntimeError when a bootstrap peer reports a different dcp_rank with the same dcp_size, which would mean miswired peer discovery.","triggerScenarios":"During PD bootstrap (_add_remote_peer / bootstrap_thread) with dcp_size == dst_dcp_size but self.dcp_rank != dst_dcp_rank — e.g. prefill DCP rank 0 bootstrapping against decode DCP rank 1.","commonSituations":"Bootstrap/metadata store has stale entries from a previous cluster layout so a prefill node pairs with the wrong decode rank; nodes restarted with different --dcp-size or rank assignments while old sessions persist; network address reuse across restarts.","solutions":["Clear stale bootstrap metadata (etcd/redis keys) and restart both PD clusters so ranks re-register cleanly","Verify both sides run identical --dcp-size and that rank assignment (e.g. CP/DCP rank derivation) is deterministic and matches","Ensure only one prefill cluster and one decode cluster share the bootstrap namespace; isolate environments otherwise"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if dcp_size == dst_dcp_size:\\n    assert dcp_rank == dst_dcp_rank, f'rank pairing mismatch {dcp_rank} vs {dst_dcp_rank}; clear bootstrap metadata'","typeGuard":null,"tryCatchPattern":"try:\\n    conn.requires_dcp_relayout(dst_dcp_size, dst_dcp_rank)\\nexcept RuntimeError:\\n    # re-bootstrap: purge stale peer entries and re-register this rank\\n    metadata_store.clear_pd_entries(); rebootstrap()","preventionTips":["Flush etcd/redis bootstrap keys between cluster restarts","Keep dcp-size identical on both sides so ranks pair 1:1","Use distinct bootstrap namespaces per environment"],"tags":["disaggregation","dcp","parallelism","bootstrap"],"backgroundTag":"rank-mismatch-parallelism","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}