sgl-project/sglang · error · NotImplementedError

NIXL heterogeneous-TP direct-to-host KV transfer is not impl

Error message

NIXL heterogeneous-TP direct-to-host KV transfer is not implemented safely yet

What it means

In the heterogeneous-TP branch of _prepare_payload_xfer (decode_tp_size != attn_tp_size, non-MLA), the destination's homogeneous memory kind must be VRAM. If it is anything else (e.g. DRAM), the code raises NotImplementedError because relayouting KV across different TP geometries directly into host memory is not yet implemented safely.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:1072

                item_len * dst_num_slots for item_len in dst_kv_item_lens
            ]
            self._init_equal_tp_prep_handle(
                peer_info.agent_name,
                dst_kv_ptrs,
                peer_info.gpu_id,
                num_slots=peer_info.dst_num_slots,
                mem_kind=dst_mem_kind,
                kv_item_lens=dst_kv_item_lens,
                kv_data_lens=dst_kv_data_lens,
                kv_xfer_lens=self.kv_args.kv_item_lens,
            )
        else:
            dst_mem_kind = _homogeneous_kv_mem_kind(
                peer_info.dst_kv_mem_kinds, "destination"
            )
            peer_info.dst_homogeneous_mem_kind = dst_mem_kind
            if dst_mem_kind != "VRAM":
                raise NotImplementedError(
                    "NIXL heterogeneous-TP direct-to-host KV transfer is not "
                    "implemented safely yet"
                )
            self._init_hetero_tp_prep_handle(
                peer_info.agent_name,
                peer_info,
                src_mem_kind=src_mem_kind,
                dst_mem_kind=dst_mem_kind,
            )

    def transfer_worker(self, queue: FastQueue, staging_buffer=None):
        # Per-worker staging strategy: lazy-created on first chunk so we
        # see kv_buffer_tensors (set by ModelRunner after engine init).
        # Never cache on self -- multiple workers would race the ring.
        staging_strategy = None

        while True:
            kv_chunk: TransferKVChunk = queue.get()

View on GitHub (pinned to 0132848349)

Solutions

1. Disable hierarchical/host KV cache on the decode instance so all dst tensors are VRAM
2. Use equal TP sizes (decode_tp == attn_tp), which takes the equal-TP path that supports other mem kinds
3. Wait for heterogeneous-TP direct-to-host support in a newer SGLang release

Example fix

# before (decode)
--tp 4 --enable-hierarchical-cache  # vs prefill attn-tp 8
# after
--tp 4   # VRAM-only dst; or match --tp 8 to take equal-TP path

hetero_tp = decode_tp_size != attn_tp_size
if hetero_tp:
    assert set(dst.kv_data_mem_kinds) == {'VRAM'}, 'hetero-TP PD requires VRAM destination'

except NotImplementedError as e:
    if 'heterogeneous-TP' in str(e):
        redeploy_decode_with_vram_only_or_equal_tp()

When it happens

Trigger: Heterogeneous-TP PD disaggregation (decode TP differs from prefill attention TP) combined with a decode-side KV cache whose memory kind resolves to non-VRAM (host/hierarchical cache enabled on decode).

Common situations: Running prefill with attn-tp 8 and decode with tp 4 while decode also enables hierarchical/HiSparse cache so KV tensors land in DRAM.

Related errors

• NIXL PD transfer does not support HiSparse combined with dec
• NIXL KV transfer has no KV memory segments
• Ascend PD transfer does not support HiSparse destination dev
• Unsupported KV cache type for decode offload
• Slice size exceeds destination token capacity for TP slice t

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3d3c9229f0fb7293. Report an issue: GitHub.