{"record":{"id":"d0bf6111ca6909b6","repo":"sgl-project/sglang","slug":"missing-nixl-destination-kv-memory-kind","errorCode":null,"errorMessage":"Missing NIXL destination KV memory kind","messagePattern":"Missing NIXL destination KV memory kind","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":1229,"sourceCode":"                            if is_dcp_transfer:\n                                kv_xfer_handle = self.send_kvcache_dcp(\n                                    req.agent_name,\n                                    src_prefill_kv_indices,\n                                    dst_info,\n                                    chunked_dst_kv_indice,\n                                    src_page_offset=kv_chunk.index_slice.start or 0,\n                                    decode_prefix_len=req.decode_prefix_len or 0,\n                                    num_kv_tokens=kv_chunk.num_kv_tokens,\n                                    notif=notif,\n                                )\n                            elif (\n                                self.is_mla_backend\n                                or self.is_hybrid_mla_backend\n                                or decode_tp_size == self.attn_tp_size\n                            ):\n                                if dst_info.kv_xfer_segments is None:\n                                    if dst_info.dst_homogeneous_mem_kind is None:\n                                        raise RuntimeError(\n                                            \"Missing NIXL destination KV memory kind\"\n                                        )\n                                    kv_xfer_handle = self.send_kvcache(\n                                        req.agent_name,\n                                        src_prefill_kv_indices,\n                                        dst_info.dst_kv_ptrs,\n                                        chunked_dst_kv_indice,\n                                        dst_info.gpu_id,\n                                        notif,\n                                        dst_mem_kind=(\n                                            dst_info.dst_homogeneous_mem_kind\n                                        ),\n                                    )\n                                else:\n                                    handles.extend(\n                                        self.send_kvcache_mixed(\n                                            req.agent_name,\n                                            src_prefill_kv_indices,","sourceCodeStart":1211,"sourceCodeEnd":1247,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L1211-L1247","documentation":"In transfer_worker, the equal-TP/MLA non-mixed path needs a destination memory kind to call send_kvcache. If the peer's kv_xfer_segments is None and dst_homogeneous_mem_kind was never populated during _prepare_payload_xfer, the runtime cannot determine where to place descriptors and raises RuntimeError — an internal invariant break, typically meaning peer registration or geometry setup diverged from the transfer path assumptions.","triggerScenarios":"A registered decode peer reaches transfer_worker on the MLA/hybrid-MLA/equal-TP path with kv_xfer_segments None while dst_homogeneous_mem_kind is also None — e.g. _prepare_payload_xfer took a path (DCP relayout, early return on n_src==0) that left the mem kind unset, or registration state was mutated between registration and transfer.","commonSituations":"Internal state divergence after partial registration, a bug in a custom backend, or racing peer re-registration; usually not a user config error.","solutions":["Check logs from _add_remote_peer/_prepare_payload_xfer for earlier exceptions or skipped branches (e.g. requires_dcp_relayout, n_src==0)","Verify prefill and decode run compatible SGLang versions and identical model/backends","Restart the PD cluster to clear inconsistent registration state","Reproduce and report as an SGLang bug with both sides' server args and registration logs"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"dst = conn.decode_kv_args_table.get(agent_name)\nassert dst is not None, 'peer not registered'\nif dst.kv_xfer_segments is None:\n    assert dst.dst_homogeneous_mem_kind is not None, 'peer missing mem kind; registration incomplete'","typeGuard":null,"tryCatchPattern":"except RuntimeError as e:\n    if 'Missing NIXL destination KV memory kind' in str(e):\n        mark_peer_stale(agent_name); re_register_peer()","preventionTips":["Monitor _add_remote_peer logs to confirm every peer completes _prepare_payload_xfer","Fail fast at registration if mem kind cannot be resolved","Restart the PD cluster after changing peer topology instead of hot-swapping"],"tags":["nixl","disaggregation","internal-invariant","transfer-worker"],"backgroundTag":"invalid-state-invariant-violation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}