sgl-project/sglang · error · ValueError

Unsupported IO backend for models with head_dim !=…

Error message

Unsupported IO backend for models with head_dim != v_head_dim: {io_backend}; expected 'kernel' or 'direct'.

What it means

Raised by the MHA host KV pool when the model uses different head_dim for K and V (head_dim != v_head_dim) and the selected IO backend is not 'kernel' or 'direct', the only backends handling asymmetric head dims.

Solutions

  1. Pass 'kernel' or 'direct'
  2. Source io_backend from server_args.hicache_io_backend instead of literals
  3. Upgrade custom fork code that used removed backend identifiers

Example fix

# before
pool.load_to_device_per_layer(dev, ..., io_backend="memcpy")
# after
pool.load_to_device_per_layer(dev, ..., io_backend="kernel")
Defensive patterns

Strategy: validation

Validate before calling

assert io_backend in ("kernel", "direct"), f"unsupported io_backend {io_backend!r}"

Type guard

def is_valid_io_backend(b: str) -> bool:
    return b in ("kernel", "direct")

Try / catch

try:
    pool.load_to_device_per_layer(...)
except ValueError as e:
    if "Unsupported IO backend" in str(e):
        pool.load_to_device_per_layer(..., io_backend="kernel")
    else:
        raise

Prevention

When it happens

Trigger: Calling load_to_device_per_layer(..., io_backend=...) with a value like 'cuda', 'memcpy', or None on the asymmetric-head-dim MLA pool.

Common situations: Passing an io_backend resolved from stale config or custom code; typos; backend names removed in newer SGLang.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ed41382b8d3a7484. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/mem_cache/pool_host/mha.py:1216

                )
            transfer_kv_per_layer_direct_pf_lf(
                src_ptrs=[self.k_buffer],
                dst_ptrs=[device_pool.k_buffer[device_layer_id]],
                src_indices=host_indices,
                dst_indices=device_indices,
                layer_id=host_layer_id,
                page_size=self.page_size,
            )
            transfer_kv_per_layer_direct_pf_lf(
                src_ptrs=[self.v_buffer],
                dst_ptrs=[device_pool.v_buffer[device_layer_id]],
                src_indices=host_indices,
                dst_indices=device_indices,
                layer_id=host_layer_id,
                page_size=self.page_size,
            )
        else:
            raise ValueError(
                f"Unsupported IO backend for models with head_dim != v_head_dim: "
                f"{io_backend}; expected 'kernel' or 'direct'."
            )

    def backup_from_device_all_layer(
        self, device_pool, host_indices, device_indices, io_backend
    ):
        (
            device_k_data_ptrs,
            device_v_data_ptrs,
            device_k_buffers,
            device_v_buffers,
        ) = self._resolve_device_transfer_buffers(device_pool)
        if io_backend == "kernel":
            if self.layout != "page_first":
                raise ValueError(
                    f"Unsupported layout for models with head_dim != v_head_dim "
                    f"and io_backend='kernel': {self.layout}; expected 'page_first'."

View on GitHub (pinned to 0132848349)