sgl-project/sglang · critical · Exception

NIXL memory registration failed for {mem_kind} kv tensors

Error message

NIXL memory registration failed for {mem_kind} kv tensors

What it means

During register_buffer_to_engine (called at init), each non-empty group of KV buffer addresses is registered with the NIXL agent via agent.register_memory(kv_addrs, mem_kind). If registration returns a falsy descriptor set for a mem_kind ('VRAM' or 'DRAM'), a generic Exception is raised. This means NIXL could not register the GPU or host buffers — typically an agent/plugin initialization problem or invalid addresses.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:1396

        ):
            kv_addrs_by_mem_kind[kv_mem_kind].append(
                (
                    kv_data_ptr,
                    kv_data_len,
                    _nixl_device_id(kv_mem_kind, self.kv_args.gpu_id),
                    "",
                )
            )
        for mem_kind in ("VRAM", "DRAM"):
            kv_addrs = kv_addrs_by_mem_kind[mem_kind]
            if not kv_addrs:
                continue
            kv_descs = self.agent.register_memory(kv_addrs, mem_kind)
            logger.debug(
                f"Register kv tensors, kind={mem_kind}, len(kv_addr)= {len(kv_addrs)}"
            )
            if not kv_descs:
                raise Exception(
                    f"NIXL memory registration failed for {mem_kind} kv tensors"
                )
            self.kv_descs.append(kv_descs)
        aux_addrs = []
        for aux_data_ptr, aux_data_len in zip(
            self.kv_args.aux_data_ptrs, self.kv_args.aux_data_lens
        ):
            aux_addrs.append((aux_data_ptr, aux_data_len, 0, ""))
        self.aux_descs = self.agent.register_memory(aux_addrs, "DRAM")
        logger.debug(f"Register aux tensors, len(aux_addrs)= {len(aux_addrs)}")
        if not self.aux_descs:
            raise Exception("NIXL memory registration failed for aux tensors")

        state_addrs = []
        for comp_ptrs, comp_lens in zip(
            self.kv_args.state_data_ptrs or [],
            self.kv_args.state_data_lens or [],
        ):

View on GitHub (pinned to 0132848349)

Solutions

1. Verify UCX/NIXL installation and that UCX_TLS supports the needed transports (e.g. UCX_TLS=rc,tcp,cuda_copy)
2. Ensure containers expose GPU and IB devices (nvidia device plugin, --device /dev/infiniband, appropriate ulimits)
3. Update nixl and ucx-py versions to ones compatible with your SGLang release
4. Log kv_addrs contents to confirm pointers/lengths are non-zero and sane before registration

def buffers_registerable(agent, addrs_by_kind):
    for kind, addrs in addrs_by_kind.items():
        if addrs and not agent.register_memory(addrs, kind):
            return False
    return True

try:
    conn.register_buffer_to_engine()
except Exception as e:
    if 'memory registration failed' in str(e):
        abort_startup_with_env_diagnostics()  # dump ucx_info, nvidia-smi, devices

When it happens

Trigger: Server startup with disaggregation enabled: kv_addrs for VRAM or DRAM are non-empty but agent.register_memory returns empty/None — e.g. the NIXL agent (UCX backend) lacks the required transport or memory access for that kind, or kv_data_ptrs/lens contain invalid values.

Common situations: Missing or misconfigured UCX/RDMA setup, NIXL plugin unavailable for CUDA memory, KV buffers allocated with an allocator NIXL cannot register, or a container without proper GPU/IB device exposure.

Related errors

• NIXL memory registration failed for aux tensors
• NIXL memory registration failed for state tensors
• NIXL KV transfer has no KV memory segments
• NIXL transfer encountered ERR room={room}
• mooncake encoder_transfer_backend requires HTTP encoders; us

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/75f97e832a3e9a92. Report an issue: GitHub.