sgl-project/sglang · critical · RuntimeError

[Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=

Error message

[Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=0x{staging_buffer.get_ptr():x}, dst_ptr=0x{dst_staging_ptr + rank_offset:x}, size={local_bytes}. The decode staging buffer may not be properly registered.

What it means

The mooncake connection's staged bulk RDMA transfer returned a non-zero code while moving decode-side KV from the staging buffer to destination slots. The message dumps src/dst pointers and size plus a hint about buffer registration.

Source

Thrown at python/sglang/srt/disaggregation/mooncake/conn.py:600

            transfer_blocks = [
                (
                    staging_buffer.get_ptr(),
                    dst_staging_ptr + rank_offset,
                    local_bytes,
                )
            ]
        else:
            transfer_blocks = [
                (
                    staging_buffer.get_ptr() + src_idx * per_layer_bytes,
                    dst_staging_ptr + rank_offset + dst_idx * per_layer_bytes,
                    per_layer_bytes,
                )
                for src_idx, dst_idx in pairs
            ]
        ret = self._transfer_data(mooncake_session_id, transfer_blocks)
        if ret != 0:
            raise RuntimeError(
                f"[Staging] Bulk RDMA transfer failed with ret={ret}. "
                f"src_ptr=0x{staging_buffer.get_ptr():x}, "
                f"dst_ptr=0x{dst_staging_ptr + rank_offset:x}, size={local_bytes}. "
                f"The decode staging buffer may not be properly registered."
            )
        return ret

    def _transfer_data(self, mooncake_session_id, transfer_blocks):
        if not transfer_blocks:
            return 0

        src_addrs, dst_addrs, lengths = zip(*transfer_blocks)
        return self.engine.batch_transfer_sync(
            mooncake_session_id, list(src_addrs), list(dst_addrs), list(lengths)
        )

    def _send_kvcache_generic(
        self,

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the decode staging buffer is registered with the mooncake engine before transfer (check registration logs)
2. Restart prefill/decode instances to re-register memory regions
3. Validate RDMA fabric health between the peers
4. Check for a version mismatch of mooncake engine between prefill and decode

catch RuntimeError matching 'Bulk RDMA transfer failed'; re-register staging buffer then retry once; else escalate room failure

When it happens

Trigger: send_kvcache_staged failing in _transfer_data: staging buffer not memory-registered with the mooncake engine, destination MR expired, or fabric fault.

Common situations: Decode engine restart invalidating MRs, buffer reuse across allocator growth without re-registration, or RDMA driver issues.

Related errors

• mooncake encoder_transfer_backend requires HTTP encoders; us
• Mooncake transfer_sync failed for {req_id} (session={session
• Rank 0 produced no embedding for {ctx.req_id}
• Mooncake Transfer Engine initialization failed.
• Failed to register buffer to Mooncake Store, error code: {re

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5dce935a66dc9c6b. Report an issue: GitHub.