sgl-project/sglang · error · ValueError

Layer-sharded MLA HiCache backup with page_first layout requ

Error message

Layer-sharded MLA HiCache backup with page_first layout requires the JIT one-layer kernel.

What it means

During layer-sharded (per-layer) MLA HiCache backup from device, the page_first layout requires a JIT-compiled one-layer transfer kernel (transfer_kv_one_layer or similar). That kernel is unavailable — typically because sgl-kernel / the JIT kernel was not built or could not be imported — so the backup cannot proceed.

Source

Thrown at python/sglang/srt/mem_cache/pool_host/mla.py:384

                else:
                    transfer_kv_per_layer_mla(
                        src=device_pool.kv_buffer[device_layer_id],
                        dst=self.kv_buffer[host_layer_id],
                        src_indices=device_indices,
                        dst_indices=host_indices,
                        item_size=self.token_stride_size,
                    )
            elif self.layout == "page_first":
                if self.can_use_jit:
                    jit_transfer_hicache_one_layer_mla(
                        cache_dst=self.data_refs[host_layer_id],
                        cache_src=device_pool.kv_buffer[device_layer_id],
                        indices_dst=host_indices,
                        indices_src=device_indices,
                        element_dim=self.kv_cache_dim,
                    )
                else:
                    raise ValueError(
                        "Layer-sharded MLA HiCache backup with page_first layout "
                        "requires the JIT one-layer kernel."
                    )
            else:
                raise ValueError(
                    f"Layer-sharded HiCache backup does not support layout: {self.layout}"
                )
        elif io_backend == "direct":
            if self.layout == "layer_first":
                transfer_kv_direct(
                    src_layers=[device_pool.kv_buffer[device_layer_id]],
                    dst_layers=[self.kv_buffer[host_layer_id]],
                    src_indices=device_indices,
                    dst_indices=host_indices,
                    page_size=self.page_size,
                )
            else:
                raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Install/rebuild sgl-kernel so the JIT one-layer transfer kernel is available (pip install sgl-kernel matching your torch/CUDA, or python -m sglang.gen_jit... / build the JIT kernels)
2. Or switch io_backend to 'direct' for the backup path, which does not need the JIT kernel
3. Or use a non-layer-sharded device pool / layer_first layout
4. Verify torch/CUDA compatibility of the installed kernels

Example fix

// before
# JIT kernel missing -> backup_from_device_all_layer raises

// after
pip install -U sgl-kernel  # rebuild so the JIT one-layer kernel exists
# or avoid the kernel path:
pool.backup_from_device_all_layer(..., io_backend="direct")

from sglang.srt.mem_cache import transfer_utils  # or the kernel module
jit_ok = _jit_one_layer_kernel_available()
if not jit_ok and layout == "page_first":
    io_backend = "direct"

def jit_one_layer_kernel_available() -> bool:
    try:
        from sglang.srt.mem_cache.transfer_utils import transfer_kv_one_layer  # noqa
        return True
    except Exception:
        return False

try:
    pool.backup_from_device_all_layer(...)
except ValueError as e:
    if "requires the JIT one-layer kernel" in str(e):
        pool.backup_from_device_all_layer(..., io_backend="direct")
    else:
        raise

When it happens

Trigger: Calling backup_from_device_all_layer / _backup_from_device_per_layer with a layer-sharded device pool and layout='page_first' when the JIT one-layer kernel is not available in the environment.

Common situations: Running a source install without compiled sgl-kernel JIT kernels; CUDA/arch mismatch preventing JIT compilation; missing kernel after an upgrade; using a container without the built extensions.

Related errors

• Unsupported layout for models with head_dim != v_head_dim an
• Unsupported layout for models with head_dim != v_head_dim an
• Unsupported IO backend for models with head_dim != v_head_di
• Unsupported layout: {self.layout}
• Unsupported IO backend: {io_backend}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7c109b54f5012182. Report an issue: GitHub.