sgl-project/sglang · error · RuntimeError

--enable-unified-memory does not support different prefill /

Error message

--enable-unified-memory does not support different prefill / decode attention TP sizes (prefill={self.attn_tp_size}, decode={dst_attn_tp_size}): the whole-envelope state cannot be TP-resliced.

What it means

With --enable-unified-memory the mamba/whole-envelope state ships as one slot-sized blob with no per-tensor dims, so it cannot be resliced across a different attention TP decomposition. _validate_envelope_kv_layout rejects differing prefill vs decode attn TP sizes before any KV is written.

Source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:789

        The unified memory pool registers ONE whole-envelope region and
        addresses the destination as ``dst_ptr + page_id * item_len`` using OUR
        ``item_len``, so a peer on a different page size / spec, or without
        unified memory, would take envelope-sized blocks at the wrong offsets.
        Must run before the first RDMA write.

        Scoped to unified memory by config, not by region count: a non-unified
        PP stage owning a single full-attention layer also registers one region,
        and `_send_kvcache_generic` pairs that with the peer by layer id.
        """
        if not get_memory().enable_unified_memory:
            return
        if dst_attn_tp_size is not None and self.attn_tp_size != dst_attn_tp_size:
            # The unified mamba state ships as one whole-slot envelope with no
            # per-tensor dims, so `_send_mamba_state_slice` cannot reslice it and
            # silently falls back to an unsliced copy. Reject here, before any KV
            # is written, rather than in `maybe_send_extra` afterwards.
            raise RuntimeError(
                "--enable-unified-memory does not support different prefill / "
                f"decode attention TP sizes (prefill={self.attn_tp_size}, "
                f"decode={dst_attn_tp_size}): the whole-envelope state cannot "
                "be TP-resliced."
            )
        src_item_lens = self.kv_args.kv_item_lens
        if (
            len(src_item_lens) != 1
            or len(dst_kv_ptrs) != 1
            or dst_kv_item_len is None
            or src_item_lens[0] != dst_kv_item_len
        ):
            raise RuntimeError(
                "PD KV layout mismatch on the whole-envelope path: prefill has "
                f"{len(src_item_lens)} KV region(s) with item_lens="
                f"{src_item_lens}, decode has {len(dst_kv_ptrs)} with item_len="
                f"{dst_kv_item_len}. With --enable-unified-memory both sides "
                "must enable it and use the same page size and model spec."

View on GitHub (pinned to 0132848349)

Solutions

Set the same attention TP size on both prefill and decode instances
Drop --enable-unified-memory if heterogeneous TP is a hard requirement
Use an MLA-backend model, which supports TP reslicing

Defensive patterns

Strategy: validation

Validate before calling

assert prefill_attn_tp == decode_attn_tp or not enable_unified_memory, "unified-memory requires equal attn TP"

Prevention

Validate TP topology in the launch script before starting PD pair
Keep a topology manifest shared by prefill and decode

When it happens

Trigger: Starting PD disaggregation with --enable-unified-memory where prefill attention-TP != decode attention-TP on a hybrid (mamba) model.

Common situations: Operators trying heterogeneous TP (e.g. prefill tp=8, decode tp=4) to right-size phases; unsupported for unified-memory hybrid models.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2d34f5162397b7e8. Report an issue: GitHub.