sgl-project/sglang · critical · Exception

NIXL memory registration failed for aux tensors

Error message

NIXL memory registration failed for aux tensors

What it means

register_buffer_to_engine registers the aux tensors (aux_data_ptrs/aux_data_lens, metadata buffers used to coordinate transfer completion) as DRAM with the NIXL agent. If aux_addrs is non-empty but agent.register_memory returns a falsy descriptor, this Exception is raised, meaning NIXL failed to register the host aux buffers.

Source

Thrown at python/sglang/srt/disaggregation/nixl/conn.py:1408

                continue
            kv_descs = self.agent.register_memory(kv_addrs, mem_kind)
            logger.debug(
                f"Register kv tensors, kind={mem_kind}, len(kv_addr)= {len(kv_addrs)}"
            )
            if not kv_descs:
                raise Exception(
                    f"NIXL memory registration failed for {mem_kind} kv tensors"
                )
            self.kv_descs.append(kv_descs)
        aux_addrs = []
        for aux_data_ptr, aux_data_len in zip(
            self.kv_args.aux_data_ptrs, self.kv_args.aux_data_lens
        ):
            aux_addrs.append((aux_data_ptr, aux_data_len, 0, ""))
        self.aux_descs = self.agent.register_memory(aux_addrs, "DRAM")
        logger.debug(f"Register aux tensors, len(aux_addrs)= {len(aux_addrs)}")
        if not self.aux_descs:
            raise Exception("NIXL memory registration failed for aux tensors")

        state_addrs = []
        for comp_ptrs, comp_lens in zip(
            self.kv_args.state_data_ptrs or [],
            self.kv_args.state_data_lens or [],
        ):
            for state_data_ptr, state_data_len in zip(comp_ptrs, comp_lens):
                if state_data_ptr == 0 or state_data_len == 0:
                    continue
                state_addrs.append(
                    (state_data_ptr, state_data_len, self.kv_args.gpu_id, "")
                )
        if state_addrs:
            self.state_descs = self.agent.register_memory(state_addrs, "VRAM")
            logger.debug(
                f"Register state tensors, len(state_addrs)= {len(state_addrs)}"
            )
            if not self.state_descs:

View on GitHub (pinned to 0132848349)

Solutions

1. Verify ucx-py/nixl versions and that host-memory (DRAM) registration works on this host
2. Check that aux_data_ptrs/aux_data_lens are populated and non-zero in KVArgsRegisterInfo
3. Re-run inside a container with adequate IPC/memory locking limits (ulimit -l) and device exposure
4. Update SGLang — auxiliary registration behavior has changed across versions

assert all(p != 0 and l > 0 for p, l in zip(kv_args.aux_data_ptrs, kv_args.aux_data_lens)), 'bad aux buffers'

try:
    conn.register_buffer_to_engine()
except Exception as e:
    if 'aux tensors' in str(e):
        verify_ucx_dram_registration(); check_ulimit_memlock()

When it happens

Trigger: Startup with aux_data_ptrs populated but register_memory(aux_addrs, 'DRAM') returns empty — host memory registration failure due to UCX/NIXL misconfiguration or invalid aux buffer addresses (zero/corrupted pointers).

Common situations: Same environmental causes as KV registration failure: broken ucx-py/nixl install, container without pinned-memory permissions or low memlock ulimit, or a model backend returning bad aux buffer pointers.

Related errors

• NIXL memory registration failed for {mem_kind} kv tensors
• NIXL memory registration failed for state tensors
• NIXL KV transfer has no KV memory segments
• NIXL PD transfer does not support HiSparse combined with dec
• NIXL heterogeneous-TP direct-to-host KV transfer is not impl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9237597f2b8ebd4c. Report an issue: GitHub.