sgl-project/sglang · critical · RuntimeError

HiSparse device KV transfer requires sgl_kernel.kvcacheio (C

Error message

HiSparse device KV transfer requires sgl_kernel.kvcacheio (CUDA/ROCm). It is not available on this backend.

What it means

hisparse_memory_pool.py imports transfer_kv_all_layer_mla from sgl_kernel.kvcacheio only when running on CUDA or HIP; on any other backend (CPU, XPU, etc.) a stub is defined that raises this RuntimeError whenever on-device MLA KV transfer is attempted (transfer_values_on_device, backup_from_device_all_layer). The fused kernel simply doesn't exist for non-CUDA/ROCm platforms.

Source

Thrown at python/sglang/srt/mem_cache/hisparse_memory_pool.py:22

from typing import Optional

import torch

from sglang.srt.layers.radix_attention import RadixAttention
from sglang.srt.mem_cache.memory_pool import DSATokenToKVPool
from sglang.srt.utils import is_cuda, is_hip

logger = logging.getLogger(__name__)

# sgl_kernel.kvcacheio is only available in CUDA/ROCm sgl-kernel builds (not XPU/MPS/NPU/CPU).
_is_cuda = is_cuda()
_is_hip = is_hip()
if _is_cuda or _is_hip:
    from sgl_kernel.kvcacheio import transfer_kv_all_layer_mla
else:

    def transfer_kv_all_layer_mla(*args, **kwargs):
        raise RuntimeError(
            "HiSparse device KV transfer requires sgl_kernel.kvcacheio (CUDA/ROCm). "
            "It is not available on this backend."
        )


class HiSparseDSATokenToKVPool(DSATokenToKVPool):
    def __init__(
        self,
        size: int,
        page_size: int,
        kv_lora_rank: int,
        dtype: torch.dtype,
        qk_rope_head_dim: int,
        layer_num: int,
        device: str,
        index_head_dim: int,
        enable_memory_saver: bool,
        kv_cache_dim: int,

View on GitHub (pinned to 0132848349)

Solutions

  1. Run on a CUDA (NVIDIA) or ROCm (AMD) GPU backend where sgl_kernel.kvcacheio is available
  2. Install/upgrade the matching sgl_kernel wheel built with kvcacheio support for your GPU platform
  3. If you're on unsupported hardware, avoid the device-transfer code paths (disable the HiSparse feature that triggers transfer_values_on_device/backup_from_device_all_layer)

Example fix

# before: running on CPU
launch_server(..., enable_hisparse=True)  # DSA on-device transfer -> RuntimeError

# after: use a CUDA machine
# CUDA_VISIBLE_DEVICES=0 launch_server(..., enable_hisparse=True)
Defensive patterns

Strategy: fallback

Validate before calling

import sglang.srt.utils as srt_utils
ok = srt_utils.is_cuda() or srt_utils.is_hip()
try:
    import sgl_kernel.kvcacheio  # noqa
except ImportError:
    ok = False
if not ok:
    raise SystemExit("HiSparse device KV transfer needs CUDA/ROCm with sgl_kernel.kvcacheio")

Type guard

def device_kv_transfer_available() -> bool:
    try:
        import sgl_kernel.kvcacheio  # noqa
        return True
    except ImportError:
        return False

Try / catch

try:
    pool.transfer_values_on_device(...)
except RuntimeError as e:
    if "sgl_kernel.kvcacheio" in str(e):
        logger.error("GPU backend required for device KV transfer; disabling this path")
        disable_device_transfer = True
    else:
        raise

Prevention

When it happens

Trigger: Running a HiSparse/DSA workload with device-side KV transfer (transfer_values_on_device or backup_from_device_all_layer paths) on hardware that is not CUDA or ROCm, or with an sgl_kernel build that lacks kvcacheio.

Common situations: Trying HiSparse + DSA/MLA on CPU or an accelerator backend; a CPU-only dev install of sgl_kernel; platform detection (is_cuda/is_hip) misidentifying the backend in custom builds.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f5ba4911cbba558a. Report an issue: GitHub.