sgl-project/sglang · error · RuntimeError

--enable-unified-memory does not support different prefill…

Error message

--enable-unified-memory does not support different prefill / decode attention TP sizes (prefill={self.attn_tp_size}, decode={dst_attn_tp_size}): the whole-envelope state cannot be TP-resliced.

What it means

With --enable-unified-memory the mamba/whole-envelope state ships as one slot-sized blob with no per-tensor dims, so it cannot be resliced across a different attention TP decomposition. _validate_envelope_kv_layout rejects differing prefill vs decode attn TP sizes before any KV is written.

Solutions

  1. Set the same attention TP size on both prefill and decode instances
  2. Drop --enable-unified-memory if heterogeneous TP is a hard requirement
  3. Use an MLA-backend model, which supports TP reslicing
Defensive patterns

Strategy: validation

Validate before calling

assert prefill_attn_tp == decode_attn_tp or not enable_unified_memory, "unified-memory requires equal attn TP"

Prevention

When it happens

Trigger: Starting PD disaggregation with --enable-unified-memory where prefill attention-TP != decode attention-TP on a hybrid (mamba) model.

Common situations: Operators trying heterogeneous TP (e.g. prefill tp=8, decode tp=4) to right-size phases; unsupported for unified-memory hybrid models.

Related errors


AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2d34f5162397b7e8. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:789

        The unified memory pool registers ONE whole-envelope region and
        addresses the destination as ``dst_ptr + page_id * item_len`` using OUR
        ``item_len``, so a peer on a different page size / spec, or without
        unified memory, would take envelope-sized blocks at the wrong offsets.
        Must run before the first RDMA write.

        Scoped to unified memory by config, not by region count: a non-unified
        PP stage owning a single full-attention layer also registers one region,
        and `_send_kvcache_generic` pairs that with the peer by layer id.
        """
        if not get_memory().enable_unified_memory:
            return
        if dst_attn_tp_size is not None and self.attn_tp_size != dst_attn_tp_size:
            # The unified mamba state ships as one whole-slot envelope with no
            # per-tensor dims, so `_send_mamba_state_slice` cannot reslice it and
            # silently falls back to an unsliced copy. Reject here, before any KV
            # is written, rather than in `maybe_send_extra` afterwards.
            raise RuntimeError(
                "--enable-unified-memory does not support different prefill / "
                f"decode attention TP sizes (prefill={self.attn_tp_size}, "
                f"decode={dst_attn_tp_size}): the whole-envelope state cannot "
                "be TP-resliced."
            )
        src_item_lens = self.kv_args.kv_item_lens
        if (
            len(src_item_lens) != 1
            or len(dst_kv_ptrs) != 1
            or dst_kv_item_len is None
            or src_item_lens[0] != dst_kv_item_len
        ):
            raise RuntimeError(
                "PD KV layout mismatch on the whole-envelope path: prefill has "
                f"{len(src_item_lens)} KV region(s) with item_lens="
                f"{src_item_lens}, decode has {len(dst_kv_ptrs)} with item_len="
                f"{dst_kv_item_len}. With --enable-unified-memory both sides "
                "must enable it and use the same page size and model spec."

View on GitHub (pinned to 0132848349)