{"record":{"id":"27b80e030c520e05","repo":"sgl-project/sglang","slug":"nixl-memory-registration-failed-for-state-tensors","errorCode":null,"errorMessage":"NIXL memory registration failed for state tensors","messagePattern":"NIXL memory registration failed for state tensors","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":1427,"sourceCode":"\n        state_addrs = []\n        for comp_ptrs, comp_lens in zip(\n            self.kv_args.state_data_ptrs or [],\n            self.kv_args.state_data_lens or [],\n        ):\n            for state_data_ptr, state_data_len in zip(comp_ptrs, comp_lens):\n                if state_data_ptr == 0 or state_data_len == 0:\n                    continue\n                state_addrs.append(\n                    (state_data_ptr, state_data_len, self.kv_args.gpu_id, \"\")\n                )\n        if state_addrs:\n            self.state_descs = self.agent.register_memory(state_addrs, \"VRAM\")\n            logger.debug(\n                f\"Register state tensors, len(state_addrs)= {len(state_addrs)}\"\n            )\n            if not self.state_descs:\n                raise Exception(\"NIXL memory registration failed for state tensors\")\n\n    def _add_remote_peer(self, decode_kv_args: KVArgsRegisterInfo):\n        agent_name = decode_kv_args.agent_name\n        if agent_name in self.decode_kv_args_table:\n            logger.info(f\"Peer {agent_name} was already registered, ignoring.\")\n            return\n        decode_kv_args.requires_dcp_relayout = self.requires_dcp_relayout(\n            decode_kv_args.dst_dcp_size, decode_kv_args.dst_dcp_rank\n        )\n        self.decode_kv_args_table[agent_name] = decode_kv_args\n        self.agent.add_remote_agent(decode_kv_args.agent_metadata)\n        if self.disaggregation_mode == DisaggregationMode.PREFILL:\n            self._prepare_payload_xfer(decode_kv_args)\n\n    def _send_kvcache_generic(\n        self,\n        peer_name: str,\n        src_data_ptrs: list[int],","sourceCodeStart":1409,"sourceCodeEnd":1445,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L1409-L1445","documentation":"When a model carries state tensors (e.g. Mamba/linear-attention state buffers via state_data_ptrs/state_data_lens), register_buffer_to_engine registers them as VRAM (after filtering zero ptr/len entries). If state_addrs is non-empty but agent.register_memory returns a falsy descriptor, this Exception is raised: NIXL could not register the GPU state buffers.","triggerScenarios":"Serving a hybrid/Mamba model with PD disaggregation where state buffers exist and their GPU registration fails — invalid state pointers, or UCX lacking CUDA memory registration support.","commonSituations":"NIXL PD transfer of hybrid linear-attention models with a ucx-py build lacking CUDA support, GPU memory/allocator issues, or version mismatch between nixl and the CUDA runtime.","solutions":["Verify UCX has CUDA memory registration support (ucx_info -d; check cuda_copy/rc transports) and matching CUDA versions","Update SGLang/nixl/ucx-py to a compatible set — hybrid-model state transfer support is actively evolving","Confirm state_data_ptrs are valid device pointers after model load (log them before registration)","If NIXL transfer for the hybrid model is unsupported in your versions, fall back to a non-disaggregated deployment or another transfer backend"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"state_addrs = [(p, l) for ptrs, lens in zip(kv_args.state_data_ptrs or [], kv_args.state_data_lens or []) for p, l in zip(ptrs, lens) if p != 0 and l != 0]\nif state_addrs:\n    assert agent.register_memory(state_addrs, 'VRAM'), 'state VRAM registration failed'","typeGuard":null,"tryCatchPattern":"try:\n    conn.register_buffer_to_engine()\nexcept Exception as e:\n    if 'state tensors' in str(e):\n        check_ucx_cuda_support(); pin_compatible_nixl_ucx_versions()","preventionTips":["For hybrid/Mamba models, verify CUDA memory registration works via ucx_info before PD deployment","Pin nixl/ucx-py/CUDA versions validated for hybrid-model state transfer","Smoke-test state buffer registration in staging before production"],"tags":["nixl","memory-registration","hybrid-model","startup","disaggregation"],"backgroundTag":"memory-registration-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}