{"record":{"id":"3d3c9229f0fb7293","repo":"sgl-project/sglang","slug":"nixl-heterogeneous-tp-direct-to-host-kv-transfer-i","errorCode":null,"errorMessage":"NIXL heterogeneous-TP direct-to-host KV transfer is not implemented safely yet","messagePattern":"NIXL heterogeneous-TP direct-to-host KV transfer is not implemented safely yet","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":1072,"sourceCode":"                item_len * dst_num_slots for item_len in dst_kv_item_lens\n            ]\n            self._init_equal_tp_prep_handle(\n                peer_info.agent_name,\n                dst_kv_ptrs,\n                peer_info.gpu_id,\n                num_slots=peer_info.dst_num_slots,\n                mem_kind=dst_mem_kind,\n                kv_item_lens=dst_kv_item_lens,\n                kv_data_lens=dst_kv_data_lens,\n                kv_xfer_lens=self.kv_args.kv_item_lens,\n            )\n        else:\n            dst_mem_kind = _homogeneous_kv_mem_kind(\n                peer_info.dst_kv_mem_kinds, \"destination\"\n            )\n            peer_info.dst_homogeneous_mem_kind = dst_mem_kind\n            if dst_mem_kind != \"VRAM\":\n                raise NotImplementedError(\n                    \"NIXL heterogeneous-TP direct-to-host KV transfer is not \"\n                    \"implemented safely yet\"\n                )\n            self._init_hetero_tp_prep_handle(\n                peer_info.agent_name,\n                peer_info,\n                src_mem_kind=src_mem_kind,\n                dst_mem_kind=dst_mem_kind,\n            )\n\n    def transfer_worker(self, queue: FastQueue, staging_buffer=None):\n        # Per-worker staging strategy: lazy-created on first chunk so we\n        # see kv_buffer_tensors (set by ModelRunner after engine init).\n        # Never cache on self -- multiple workers would race the ring.\n        staging_strategy = None\n\n        while True:\n            kv_chunk: TransferKVChunk = queue.get()","sourceCodeStart":1054,"sourceCodeEnd":1090,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L1054-L1090","documentation":"In the heterogeneous-TP branch of _prepare_payload_xfer (decode_tp_size != attn_tp_size, non-MLA), the destination's homogeneous memory kind must be VRAM. If it is anything else (e.g. DRAM), the code raises NotImplementedError because relayouting KV across different TP geometries directly into host memory is not yet implemented safely.","triggerScenarios":"Heterogeneous-TP PD disaggregation (decode TP differs from prefill attention TP) combined with a decode-side KV cache whose memory kind resolves to non-VRAM (host/hierarchical cache enabled on decode).","commonSituations":"Running prefill with attn-tp 8 and decode with tp 4 while decode also enables hierarchical/HiSparse cache so KV tensors land in DRAM.","solutions":["Disable hierarchical/host KV cache on the decode instance so all dst tensors are VRAM","Use equal TP sizes (decode_tp == attn_tp), which takes the equal-TP path that supports other mem kinds","Wait for heterogeneous-TP direct-to-host support in a newer SGLang release"],"exampleFix":"# before (decode)\n--tp 4 --enable-hierarchical-cache  # vs prefill attn-tp 8\n# after\n--tp 4   # VRAM-only dst; or match --tp 8 to take equal-TP path","handlingStrategy":"validation","validationCode":"hetero_tp = decode_tp_size != attn_tp_size\nif hetero_tp:\n    assert set(dst.kv_data_mem_kinds) == {'VRAM'}, 'hetero-TP PD requires VRAM destination'","typeGuard":null,"tryCatchPattern":"except NotImplementedError as e:\n    if 'heterogeneous-TP' in str(e):\n        redeploy_decode_with_vram_only_or_equal_tp()","preventionTips":["Disable hierarchical cache on decode when TP geometries differ","Prefer equal TP sizes across PD nodes","Track the hetero-TP PD support matrix in release notes"],"tags":["nixl","disaggregation","heterogeneous-tp","hicache","pd-disaggregation"],"backgroundTag":"unsupported-feature-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}