sgl-project/sglang · error · RuntimeError

Missing NIXL destination KV memory kind

Error message

Missing NIXL destination KV memory kind

What it means

In transfer_worker, the equal-TP/MLA non-mixed path needs a destination memory kind to call send_kvcache. If the peer's kv_xfer_segments is None and dst_homogeneous_mem_kind was never populated during _prepare_payload_xfer, the runtime cannot determine where to place descriptors and raises RuntimeError — an internal invariant break, typically meaning peer registration or geometry setup diverged from the transfer path assumptions.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:1229

                            if is_dcp_transfer:
                                kv_xfer_handle = self.send_kvcache_dcp(
                                    req.agent_name,
                                    src_prefill_kv_indices,
                                    dst_info,
                                    chunked_dst_kv_indice,
                                    src_page_offset=kv_chunk.index_slice.start or 0,
                                    decode_prefix_len=req.decode_prefix_len or 0,
                                    num_kv_tokens=kv_chunk.num_kv_tokens,
                                    notif=notif,
                                )
                            elif (
                                self.is_mla_backend
                                or self.is_hybrid_mla_backend
                                or decode_tp_size == self.attn_tp_size
                            ):
                                if dst_info.kv_xfer_segments is None:
                                    if dst_info.dst_homogeneous_mem_kind is None:
                                        raise RuntimeError(
                                            "Missing NIXL destination KV memory kind"
                                        )
                                    kv_xfer_handle = self.send_kvcache(
                                        req.agent_name,
                                        src_prefill_kv_indices,
                                        dst_info.dst_kv_ptrs,
                                        chunked_dst_kv_indice,
                                        dst_info.gpu_id,
                                        notif,
                                        dst_mem_kind=(
                                            dst_info.dst_homogeneous_mem_kind
                                        ),
                                    )
                                else:
                                    handles.extend(
                                        self.send_kvcache_mixed(
                                            req.agent_name,
                                            src_prefill_kv_indices,

View on GitHub (pinned to 0132848349)

Solutions

1. Check logs from _add_remote_peer/_prepare_payload_xfer for earlier exceptions or skipped branches (e.g. requires_dcp_relayout, n_src==0)
2. Verify prefill and decode run compatible SGLang versions and identical model/backends
3. Restart the PD cluster to clear inconsistent registration state
4. Reproduce and report as an SGLang bug with both sides' server args and registration logs

dst = conn.decode_kv_args_table.get(agent_name)
assert dst is not None, 'peer not registered'
if dst.kv_xfer_segments is None:
    assert dst.dst_homogeneous_mem_kind is not None, 'peer missing mem kind; registration incomplete'

except RuntimeError as e:
    if 'Missing NIXL destination KV memory kind' in str(e):
        mark_peer_stale(agent_name); re_register_peer()

When it happens

Trigger: A registered decode peer reaches transfer_worker on the MLA/hybrid-MLA/equal-TP path with kv_xfer_segments None while dst_homogeneous_mem_kind is also None — e.g. _prepare_payload_xfer took a path (DCP relayout, early return on n_src==0) that left the mem kind unset, or registration state was mutated between registration and transfer.

Common situations: Internal state divergence after partial registration, a bug in a custom backend, or racing peer re-registration; usually not a user config error.

Related errors

• Missing aux index for last chunk
• Unexpected compressed-MLA dst_kv_ptrs length {len(dst_kv_ptr
• NIXL PD transfer does not support HiSparse combined with dec
• NIXL KV transfer has no KV memory segments
• NIXL heterogeneous-TP direct-to-host KV transfer is not impl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d0bf6111ca6909b6. Report an issue: GitHub.