sgl-project/sglang · error · ValueError

Layer-sharded HiCache backup does not support IO backend: {i

Error message

Layer-sharded HiCache backup does not support IO backend: {io_backend}

What it means

_backup_from_device_per_layer accepts only io_backend values "kernel" and "direct". Any other value (e.g. "kernel_ascend", "vanilla", a typo, or None) reaches the final else and raises, because no per-layer implementation exists for that IO backend.

Source

Thrown at python/sglang/srt/mem_cache/pool_host/mla.py:407

                raise ValueError(
                    f"Layer-sharded HiCache backup does not support layout: {self.layout}"
                )
        elif io_backend == "direct":
            if self.layout == "layer_first":
                transfer_kv_direct(
                    src_layers=[device_pool.kv_buffer[device_layer_id]],
                    dst_layers=[self.kv_buffer[host_layer_id]],
                    src_indices=device_indices,
                    dst_indices=host_indices,
                    page_size=self.page_size,
                )
            else:
                raise ValueError(
                    "Layer-sharded direct HiCache backup only supports "
                    f"layer_first layout, got {self.layout}"
                )
        else:
            raise ValueError(
                f"Layer-sharded HiCache backup does not support IO backend: {io_backend}"
            )

    def _resolve_device_transfer_buffers(self, device_pool):
        if self.mtp_draft_device_pools:
            return self.packed_device_data_ptrs, self.packed_device_kv_buffers
        return device_pool.data_ptrs, device_pool.kv_buffer

    def backup_from_device_all_layer(
        self, device_pool, host_indices, device_indices, io_backend
    ):
        host_indices = self.maybe_dcp_kernel_indices(host_indices)
        device_indices = self.maybe_dcp_kernel_indices(device_indices)
        if self._is_device_layer_sharded(device_pool):
            for layer_id in self._owned_device_layer_ids(device_pool):
                self._backup_from_device_per_layer(
                    device_pool, host_indices, device_indices, layer_id, io_backend
                )

View on GitHub (pinned to 0132848349)

Solutions

1. Use io_backend="kernel" or "direct" for layer-sharded configurations
2. Fix or validate the --hicache-io-backend server argument before startup
3. Disable layer sharding so the all-layer dispatch (which supports kernel_ascend) is used

Example fix

# before
server_args.hicache_io_backend = "kernel_ascend"  # with CP layer sharding
# after
server_args.hicache_io_backend = "kernel"

VALID_BACKENDS = {"kernel", "direct"}
if io_backend not in VALID_BACKENDS:
    raise ConfigError(f"io_backend must be one of {VALID_BACKENDS}, got {io_backend!r}")

def is_valid_per_layer_backend(b: str) -> bool:
    return b in ("kernel", "direct")

try:
    pool.backup_from_device_all_layer(dp, hi, di, io_backend)
except ValueError as e:
    if "does not support IO backend" in str(e):
        io_backend = "kernel"
        pool.backup_from_device_all_layer(dp, hi, di, io_backend)
    else:
        raise

When it happens

Trigger: backup_from_device_all_layer receives an io_backend string outside {kernel, direct} while the device pool is layer-sharded, so the per-layer loop is taken and the backend dispatch fails.

Common situations: Setting --hicache-io-backend kernel_ascend (or an unset/misspelled value) on a CP-layer-sharded deployment; a new IO backend added to backup_from_device_all_layer without a per-layer counterpart.

Related errors

• Unsupported V4 paged host layout/backend: {self.layout}/{io_
• Unsupported layout for models with head_dim != v_head_dim an
• Unsupported layout for models with head_dim != v_head_dim an
• Unsupported IO backend for models with head_dim != v_head_di
• Unsupported IO backend: {io_backend}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/69e2ac913b2dc9b6. Report an issue: GitHub.