sgl-project/sglang · critical · RuntimeError

PD DCP source/destination KV geometry differs: src={src_toke

Error message

PD DCP source/destination KV geometry differs: src={src_token_lens}, dst={dst_token_lens}

What it means

Before DCP transfer, prepare_dcp_token_item_lens converts both sides' per-page item lengths into page counts and requires identical KV geometry (same number of tokens per page sequence). If prefill and decode allocated pages with different token lengths, transfer would corrupt KV, so it raises RuntimeError.

Source

Thrown at python/sglang/srt/disaggregation/common/conn.py:336

        if (
            self.dcp_size == 1
            and dst_dcp_size > 1
            and (self.is_mla_backend or self.is_hybrid_mla_backend)
        ):
            return True

        raise RuntimeError(
            f"Unsupported PD DCP topology: {self.dcp_size} -> {dst_dcp_size}"
        )

    def prepare_dcp_token_item_lens(self, dst_page_item_lens: List[int]) -> List[int]:
        page_size = self.kv_args.page_size
        src_token_lens = [
            item_len // page_size for item_len in self.kv_args.kv_item_lens
        ]
        dst_token_lens = [item_len // page_size for item_len in dst_page_item_lens]
        if src_token_lens != dst_token_lens:
            raise RuntimeError(
                "PD DCP source/destination KV geometry differs: "
                f"src={src_token_lens}, dst={dst_token_lens}"
            )
        return src_token_lens

    def check_status(self, bootstrap_room: int) -> KVPoll:
        return self.request_status[bootstrap_room]

    def update_status(self, bootstrap_room: int, status: KVPoll):
        if bootstrap_room not in self.request_status:
            # Do not resurrect a cleared entry with Failed: once clear() has
            # popped the room from request_status, any late update_status(Failed)
            # (e.g. from abort()) must be a no-op. Otherwise a Failed entry could
            # pollute a future request that reuses the same bootstrap_room.
            if status == KVPoll.Failed:
                return
            self.request_status[bootstrap_room] = status
        else:

View on GitHub (pinned to 0132848349)

Solutions

Run identical sglang versions and matching --page-size, chunked-prefill-size, and batch-composition-relevant settings on both PD sides
Clear stale bootstrap metadata and restart so prefill/decode ranks pair fresh with consistent batches
Verify request routing so each prefill batch maps to the corresponding decode allocation

Defensive patterns

Strategy: validation

Validate before calling

src = [l // page_size for l in kv_args.kv_item_lens]\ndst = [l // page_size for l in dst_page_item_lens]\nassert src == dst, f'KV geometry differs {src} vs {dst}; align chunking/page settings'

Try / catch

try:\n    conn.prepare_dcp_token_item_lens(dst_page_item_lens)\nexcept RuntimeError:\n    # re-pair this batch with the matching decode allocation / restart PD cleanly

Prevention

Use identical sglang versions and page/chunked-prefill settings on both sides
Restart both PD clusters together after any scheduler config change
Avoid mixing stale decode ranks with fresh prefill ranks; clear bootstrap metadata on redeploy

When it happens

Trigger: During bootstrap/payload transfer when [item_len // page_size for item_len in kv_args.kv_item_lens] on the prefill side differs from the decode side's dst_page_item_lens — i.e. batch composition or page allocation differs between the paired PD ranks.

Common situations: Prefill and decode chunked-prefill sizes / max running requests differ so batches split into different token-length groups; page_size mismatch between sides; scheduler changes on one side only (version skew between prefill and decode images); a decode rank paired with a prefill rank serving a different request set after stale bootstrap metadata.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e5d14d89baab6914. Report an issue: GitHub.