sgl-project/sglang · error · RuntimeError

[Staging] KV transfer via staging buffer failed: {e}. sessio

Error message

[Staging] KV transfer via staging buffer failed: {e}. session={session_id}

What it means

DecodeStagingHandler.transfer wraps the underlying staging-buffer copy/transfer call and re-raises any exception as RuntimeError with the session id. It is a boundary wrapper: the root cause is in the chained exception (the original 'e'), typically a CUDA error, shape/dtype mismatch, or NCCL/RDMA failure inside the staging transfer.

Source

Thrown at python/sglang/srt/disaggregation/common/staging_handler.py:632

    ) -> int:
        """Execute staged transfer (gather + RDMA).

        Returns 0 on success, -1 to signal fallback to slice path.
        """
        try:
            return self.kv_manager.send_kvcache_staged(
                session_id,
                prefill_kv_indices,
                dst_staging_ptr,
                dst_staging_size,
                target_info.dst_tp_rank,
                target_info.dst_attn_tp_size,
                target_info.dst_kv_item_len,
                target_info.dst_kv_layer_ids,
                staging_buffer=self.staging_buffer,
            )
        except Exception as e:
            raise RuntimeError(
                f"[Staging] KV transfer via staging buffer failed: {e}. "
                f"session={session_id}"
            ) from e


def _get_custom_mem_pool(device: str):
    """Get custom memory pool for staging buffer allocation (backend-agnostic).

    Returns (custom_mem_pool, pool_type) tuple. custom_mem_pool may be None
    if no custom pool is configured.
    """
    from sglang.srt.disaggregation.mooncake.utils import (
        init_mooncake_custom_mem_pool,
    )

    _, custom_mem_pool, pool_type = init_mooncake_custom_mem_pool(device)
    if custom_mem_pool is None:
        logger.info(

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect the chained cause (raise ... from e) — the actual failing operation is the original exception, not this wrapper
2. Verify sender and receiver agree on attn_tp_size, kv_item_len and kv_layer_ids (target_info fields shown at the call site)
3. Check dmesg/CUDA logs for device errors and NCCL logs for transport failures; retry the request/session after fixing the underlying mismatch

assert target_info.dst_attn_tp_size == sender_tp_size and target_info.dst_kv_item_len == expected_item_len

try:
    handler.transfer(...)
except RuntimeError as e:
    if '[Staging]' in str(e) and e.__cause__:
        logger.error('staging transfer failed (session=%s): %s', session_id, e.__cause__)
    raise

When it happens

Trigger: Any exception thrown by the staging transfer call for a session — mismatched dst_kv_item_len/layer ids between sender and receiver, invalid device pointers, CUDA OOM or IPC handle failures, desynchronized prefill/decode buffer sizes.

Common situations: Prefill and decode servers built with different tensor-parallel sizes or layer counts; transient NCCL/RDMA faults; race where the decode buffer was reallocated mid-session.

Related errors

• Cannot resolve total_kv_heads: kv_args has neither total_kv_
• Staging is enabled but kv_manager._staging_ctx.allocator is
• Staging is enabled but kv_manager.kv_buffer_tensors is None.
• SGLANG_DISAGG_STAGING_BUFFER is designed for non-MLA models
• NIXL transfer encountered ERR room={room}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f4b09190086192a5. Report an issue: GitHub.