{"record":{"id":"e5d14d89baab6914","repo":"sgl-project/sglang","slug":"pd-dcp-source-destination-kv-geometry-differs-src","errorCode":null,"errorMessage":"PD DCP source/destination KV geometry differs: src={src_token_lens}, dst={dst_token_lens}","messagePattern":"PD DCP source/destination KV geometry differs: src=(.+?), dst=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/common/conn.py","lineNumber":336,"sourceCode":"        if (\n            self.dcp_size == 1\n            and dst_dcp_size > 1\n            and (self.is_mla_backend or self.is_hybrid_mla_backend)\n        ):\n            return True\n\n        raise RuntimeError(\n            f\"Unsupported PD DCP topology: {self.dcp_size} -> {dst_dcp_size}\"\n        )\n\n    def prepare_dcp_token_item_lens(self, dst_page_item_lens: List[int]) -> List[int]:\n        page_size = self.kv_args.page_size\n        src_token_lens = [\n            item_len // page_size for item_len in self.kv_args.kv_item_lens\n        ]\n        dst_token_lens = [item_len // page_size for item_len in dst_page_item_lens]\n        if src_token_lens != dst_token_lens:\n            raise RuntimeError(\n                \"PD DCP source/destination KV geometry differs: \"\n                f\"src={src_token_lens}, dst={dst_token_lens}\"\n            )\n        return src_token_lens\n\n    def check_status(self, bootstrap_room: int) -> KVPoll:\n        return self.request_status[bootstrap_room]\n\n    def update_status(self, bootstrap_room: int, status: KVPoll):\n        if bootstrap_room not in self.request_status:\n            # Do not resurrect a cleared entry with Failed: once clear() has\n            # popped the room from request_status, any late update_status(Failed)\n            # (e.g. from abort()) must be a no-op. Otherwise a Failed entry could\n            # pollute a future request that reuses the same bootstrap_room.\n            if status == KVPoll.Failed:\n                return\n            self.request_status[bootstrap_room] = status\n        else:","sourceCodeStart":318,"sourceCodeEnd":354,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/common/conn.py#L318-L354","documentation":"Before DCP transfer, prepare_dcp_token_item_lens converts both sides' per-page item lengths into page counts and requires identical KV geometry (same number of tokens per page sequence). If prefill and decode allocated pages with different token lengths, transfer would corrupt KV, so it raises RuntimeError.","triggerScenarios":"During bootstrap/payload transfer when [item_len // page_size for item_len in kv_args.kv_item_lens] on the prefill side differs from the decode side's dst_page_item_lens — i.e. batch composition or page allocation differs between the paired PD ranks.","commonSituations":"Prefill and decode chunked-prefill sizes / max running requests differ so batches split into different token-length groups; page_size mismatch between sides; scheduler changes on one side only (version skew between prefill and decode images); a decode rank paired with a prefill rank serving a different request set after stale bootstrap metadata.","solutions":["Run identical sglang versions and matching --page-size, chunked-prefill-size, and batch-composition-relevant settings on both PD sides","Clear stale bootstrap metadata and restart so prefill/decode ranks pair fresh with consistent batches","Verify request routing so each prefill batch maps to the corresponding decode allocation"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"src = [l // page_size for l in kv_args.kv_item_lens]\\ndst = [l // page_size for l in dst_page_item_lens]\\nassert src == dst, f'KV geometry differs {src} vs {dst}; align chunking/page settings'","typeGuard":null,"tryCatchPattern":"try:\\n    conn.prepare_dcp_token_item_lens(dst_page_item_lens)\\nexcept RuntimeError:\\n    # re-pair this batch with the matching decode allocation / restart PD cleanly","preventionTips":["Use identical sglang versions and page/chunked-prefill settings on both sides","Restart both PD clusters together after any scheduler config change","Avoid mixing stale decode ranks with fresh prefill ranks; clear bootstrap metadata on redeploy"],"tags":["disaggregation","dcp","kv-cache","geometry-mismatch"],"backgroundTag":"kv-layout-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}