sgl-project/sglang · error · ValueError

Unsupported layout for models with head_dim != v_head_dim an

Error message

Unsupported layout for models with head_dim != v_head_dim and io_backend='kernel': {self.layout}; expected 'page_first'.

What it means

For MLA models with head_dim != v_head_dim, load_to_device_per_layer with io_backend='kernel' uses transfer_kv_per_layer_mla_pf_lf which requires the host buffer in 'page_first' layout so its kernel indexing is valid.

Source

Thrown at python/sglang/srt/mem_cache/pool_host/mha.py:1170

        host_indices,
        device_indices,
        layer_id,
        io_backend,
        *,
        is_draft: bool = False,
    ):
        if self.device_pool is not None:
            if not is_draft and not self._is_device_layer_owned(device_pool, layer_id):
                return
            # MTP draft layers do not participate in CP layer sharding.
            host_layer_id = layer_id if is_draft else self._host_layer_index(layer_id)
            device_layer_id = 0 if is_draft else layer_id
        else:
            host_layer_id = device_layer_id = layer_id

        if io_backend == "kernel":
            if self.layout != "page_first":
                raise ValueError(
                    f"Unsupported layout for models with head_dim != v_head_dim "
                    f"and io_backend='kernel': {self.layout}; expected 'page_first'."
                )
            transfer_kv_per_layer_mla_pf_lf(
                src=self.k_buffer,
                dst=device_pool.k_buffer[device_layer_id],
                src_indices=host_indices,
                dst_indices=device_indices,
                layer_id=host_layer_id,
                item_size=self._k_token_stride_size(),
                src_layout_dim=self._k_layout_dim(),
            )
            transfer_kv_per_layer_mla_pf_lf(
                src=self.v_buffer,
                dst=device_pool.v_buffer[device_layer_id],
                src_indices=host_indices,
                dst_indices=device_indices,
                layer_id=host_layer_id,

View on GitHub (pinned to 0132848349)

Solutions

  1. Set the host pool layout to 'page_first' when using io_backend='kernel'
  2. Or switch io_backend to 'direct' to use a page_first_direct pool
  3. Align hicache_io_backend and host pool layout in server args

Example fix

# before
pool.load_to_device_per_layer(dev, layer_id=0, host_indices=h, device_indices=d, io_backend="kernel")  # layout == "page_first_direct"
# after
pool.load_to_device_per_layer(dev, layer_id=0, host_indices=h, device_indices=d, io_backend="direct")
Defensive patterns

Strategy: validation

Validate before calling

if io_backend == "kernel":
    assert pool.layout == "page_first", "kernel backend requires page_first for asymmetric head dims"

Type guard

def kernel_backend_compatible(pool, io_backend: bool) -> bool:
    return io_backend != "kernel" or pool.layout == "page_first"

Try / catch

try:
    pool.load_to_device_per_layer(...)
except ValueError as e:
    if "expected 'page_first'" in str(e):
        rebuild pool with layout="page_first", or retry with io_backend="direct"
    raise

Prevention

When it happens

Trigger: Calling load_to_device_per_layer(..., io_backend='kernel') on a head_dim!=v_head_dim host pool whose layout is 'page_first_direct' or anything else.

Common situations: Setting hicache io_backend to 'kernel' while the host pool was allocated with page_first_direct layout (mismatched config); mixing direct-layout storage files with kernel backend.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0660b3bc1c17ad3b. Report an issue: GitHub.