{"record":{"id":"f4b09190086192a5","repo":"sgl-project/sglang","slug":"staging-kv-transfer-via-staging-buffer-failed","errorCode":null,"errorMessage":"[Staging] KV transfer via staging buffer failed: {e}. session={session_id}","messagePattern":"\\[Staging\\] KV transfer via staging buffer failed: (.+?)\\. session=(.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/disaggregation/common/staging_handler.py","lineNumber":632,"sourceCode":"    ) -> int:\n        \"\"\"Execute staged transfer (gather + RDMA).\n\n        Returns 0 on success, -1 to signal fallback to slice path.\n        \"\"\"\n        try:\n            return self.kv_manager.send_kvcache_staged(\n                session_id,\n                prefill_kv_indices,\n                dst_staging_ptr,\n                dst_staging_size,\n                target_info.dst_tp_rank,\n                target_info.dst_attn_tp_size,\n                target_info.dst_kv_item_len,\n                target_info.dst_kv_layer_ids,\n                staging_buffer=self.staging_buffer,\n            )\n        except Exception as e:\n            raise RuntimeError(\n                f\"[Staging] KV transfer via staging buffer failed: {e}. \"\n                f\"session={session_id}\"\n            ) from e\n\n\ndef _get_custom_mem_pool(device: str):\n    \"\"\"Get custom memory pool for staging buffer allocation (backend-agnostic).\n\n    Returns (custom_mem_pool, pool_type) tuple. custom_mem_pool may be None\n    if no custom pool is configured.\n    \"\"\"\n    from sglang.srt.disaggregation.mooncake.utils import (\n        init_mooncake_custom_mem_pool,\n    )\n\n    _, custom_mem_pool, pool_type = init_mooncake_custom_mem_pool(device)\n    if custom_mem_pool is None:\n        logger.info(","sourceCodeStart":614,"sourceCodeEnd":650,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/common/staging_handler.py#L614-L650","documentation":"DecodeStagingHandler.transfer wraps the underlying staging-buffer copy/transfer call and re-raises any exception as RuntimeError with the session id. It is a boundary wrapper: the root cause is in the chained exception (the original 'e'), typically a CUDA error, shape/dtype mismatch, or NCCL/RDMA failure inside the staging transfer.","triggerScenarios":"Any exception thrown by the staging transfer call for a session — mismatched dst_kv_item_len/layer ids between sender and receiver, invalid device pointers, CUDA OOM or IPC handle failures, desynchronized prefill/decode buffer sizes.","commonSituations":"Prefill and decode servers built with different tensor-parallel sizes or layer counts; transient NCCL/RDMA faults; race where the decode buffer was reallocated mid-session.","solutions":["Inspect the chained cause (raise ... from e) — the actual failing operation is the original exception, not this wrapper","Verify sender and receiver agree on attn_tp_size, kv_item_len and kv_layer_ids (target_info fields shown at the call site)","Check dmesg/CUDA logs for device errors and NCCL logs for transport failures; retry the request/session after fixing the underlying mismatch"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"assert target_info.dst_attn_tp_size == sender_tp_size and target_info.dst_kv_item_len == expected_item_len","typeGuard":null,"tryCatchPattern":"try:\n    handler.transfer(...)\nexcept RuntimeError as e:\n    if '[Staging]' in str(e) and e.__cause__:\n        logger.error('staging transfer failed (session=%s): %s', session_id, e.__cause__)\n    raise","preventionTips":["Always inspect __cause__ of the wrapper error","Validate dst target_info fields before initiating transfer"],"tags":["disaggregation","staging-buffer","transfer-failure","wrapper-exception"],"backgroundTag":"kv-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}