{"record":{"id":"75f97e832a3e9a92","repo":"sgl-project/sglang","slug":"nixl-memory-registration-failed-for-mem-kind-kv","errorCode":null,"errorMessage":"NIXL memory registration failed for {mem_kind} kv tensors","messagePattern":"NIXL memory registration failed for (.+?) kv tensors","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/nixl/conn.py","lineNumber":1396,"sourceCode":"        ):\n            kv_addrs_by_mem_kind[kv_mem_kind].append(\n                (\n                    kv_data_ptr,\n                    kv_data_len,\n                    _nixl_device_id(kv_mem_kind, self.kv_args.gpu_id),\n                    \"\",\n                )\n            )\n        for mem_kind in (\"VRAM\", \"DRAM\"):\n            kv_addrs = kv_addrs_by_mem_kind[mem_kind]\n            if not kv_addrs:\n                continue\n            kv_descs = self.agent.register_memory(kv_addrs, mem_kind)\n            logger.debug(\n                f\"Register kv tensors, kind={mem_kind}, len(kv_addr)= {len(kv_addrs)}\"\n            )\n            if not kv_descs:\n                raise Exception(\n                    f\"NIXL memory registration failed for {mem_kind} kv tensors\"\n                )\n            self.kv_descs.append(kv_descs)\n        aux_addrs = []\n        for aux_data_ptr, aux_data_len in zip(\n            self.kv_args.aux_data_ptrs, self.kv_args.aux_data_lens\n        ):\n            aux_addrs.append((aux_data_ptr, aux_data_len, 0, \"\"))\n        self.aux_descs = self.agent.register_memory(aux_addrs, \"DRAM\")\n        logger.debug(f\"Register aux tensors, len(aux_addrs)= {len(aux_addrs)}\")\n        if not self.aux_descs:\n            raise Exception(\"NIXL memory registration failed for aux tensors\")\n\n        state_addrs = []\n        for comp_ptrs, comp_lens in zip(\n            self.kv_args.state_data_ptrs or [],\n            self.kv_args.state_data_lens or [],\n        ):","sourceCodeStart":1378,"sourceCodeEnd":1414,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/nixl/conn.py#L1378-L1414","documentation":"During register_buffer_to_engine (called at init), each non-empty group of KV buffer addresses is registered with the NIXL agent via agent.register_memory(kv_addrs, mem_kind). If registration returns a falsy descriptor set for a mem_kind ('VRAM' or 'DRAM'), a generic Exception is raised. This means NIXL could not register the GPU or host buffers — typically an agent/plugin initialization problem or invalid addresses.","triggerScenarios":"Server startup with disaggregation enabled: kv_addrs for VRAM or DRAM are non-empty but agent.register_memory returns empty/None — e.g. the NIXL agent (UCX backend) lacks the required transport or memory access for that kind, or kv_data_ptrs/lens contain invalid values.","commonSituations":"Missing or misconfigured UCX/RDMA setup, NIXL plugin unavailable for CUDA memory, KV buffers allocated with an allocator NIXL cannot register, or a container without proper GPU/IB device exposure.","solutions":["Verify UCX/NIXL installation and that UCX_TLS supports the needed transports (e.g. UCX_TLS=rc,tcp,cuda_copy)","Ensure containers expose GPU and IB devices (nvidia device plugin, --device /dev/infiniband, appropriate ulimits)","Update nixl and ucx-py versions to ones compatible with your SGLang release","Log kv_addrs contents to confirm pointers/lengths are non-zero and sane before registration"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"def buffers_registerable(agent, addrs_by_kind):\n    for kind, addrs in addrs_by_kind.items():\n        if addrs and not agent.register_memory(addrs, kind):\n            return False\n    return True","typeGuard":null,"tryCatchPattern":"try:\n    conn.register_buffer_to_engine()\nexcept Exception as e:\n    if 'memory registration failed' in str(e):\n        abort_startup_with_env_diagnostics()  # dump ucx_info, nvidia-smi, devices","preventionTips":["Pre-flight UCX/NIXL with a small register/transfer smoke test before launching the server","Pin compatible nixl/ucx-py versions in the deployment image","Expose GPU and IB devices correctly in containers; verify with ucx_info -d"],"tags":["nixl","memory-registration","startup","rdma","disaggregation"],"backgroundTag":"memory-registration-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}