sgl-project/sglang · error · RuntimeError

PD KV layout mismatch on the whole-envelope path: prefill ha

Error message

PD KV layout mismatch on the whole-envelope path: prefill has {len(src_item_lens)} KV region(s) with item_lens={src_item_lens}, decode has {len(dst_kv_ptrs)} with item_len={dst_kv_item_len}. With --enable-unified-memory both sides must enable it and use the same page size and model spec.

What it means

On the whole-envelope path (unified memory), the layout validation requires exactly one KV region on each side with identical item_len. Any deviation — multiple regions, missing dst item_len, or size mismatch — means the two sides disagree on page size or model spec and the transfer is rejected.

Source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:802

        if dst_attn_tp_size is not None and self.attn_tp_size != dst_attn_tp_size:
            # The unified mamba state ships as one whole-slot envelope with no
            # per-tensor dims, so `_send_mamba_state_slice` cannot reslice it and
            # silently falls back to an unsliced copy. Reject here, before any KV
            # is written, rather than in `maybe_send_extra` afterwards.
            raise RuntimeError(
                "--enable-unified-memory does not support different prefill / "
                f"decode attention TP sizes (prefill={self.attn_tp_size}, "
                f"decode={dst_attn_tp_size}): the whole-envelope state cannot "
                "be TP-resliced."
            )
        src_item_lens = self.kv_args.kv_item_lens
        if (
            len(src_item_lens) != 1
            or len(dst_kv_ptrs) != 1
            or dst_kv_item_len is None
            or src_item_lens[0] != dst_kv_item_len
        ):
            raise RuntimeError(
                "PD KV layout mismatch on the whole-envelope path: prefill has "
                f"{len(src_item_lens)} KV region(s) with item_lens="
                f"{src_item_lens}, decode has {len(dst_kv_ptrs)} with item_len="
                f"{dst_kv_item_len}. With --enable-unified-memory both sides "
                "must enable it and use the same page size and model spec."
            )

    def _await_transfer_futures(self, futures) -> int:
        """Await a chunk's per-layer RDMA writes; return the first non-zero status.
        cancel() is a no-op for a running future, so with deferred release on we
        still drain the running ones before returning (no write may outlive this
        call, which the drain-ack relies on). Off: original early-return."""
        ret = 0
        for future in concurrent.futures.as_completed(futures):
            try:
                status = future.result()
            except concurrent.futures.CancelledError:
                continue

View on GitHub (pinned to 0132848349)

Solutions

Enable --enable-unified-memory on BOTH prefill and decode
Use identical --page-size on both sides
Confirm both sides load the same model spec (same checkpoint/config)
Re-check custom weight loading that alters kv_item_lens

Defensive patterns

Strategy: validation

Validate before calling

assert page_size_prefill == page_size_decode
assert unified_memory_prefill == unified_memory_decode

Prevention

Templatize PD launch configs so page-size/model flags cannot drift
Smoke-test one request after PD bring-up

When it happens

Trigger: Prefill and decode disagreeing on page_size / model / unified-memory flag, so kv_item_lens and destination descriptors don't line up one-to-one.

Common situations: One side launched without --enable-unified-memory, different --page-size flags, or different model revisions between PD instances.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2e31a27b0b9d2dac. Report an issue: GitHub.