sgl-project/sglang · error · ValueError

Unsupported IO backend

Error message

Unsupported IO backend: {io_backend}

What it means

Terminal dispatch guard in load_to_device_per_layer: the MHA host pool implements H2D restore only for specific io_backends ('kernel'-family, 'direct', 'kernel_ascend'). Any other backend string (or a typo) has no transfer implementation and the method refuses to proceed.

Solutions

1. Set io_backend to one of the supported values for hierarchical cache ('kernel', 'direct', 'kernel_ascend' as appropriate for your device)
2. Disable hierarchical cache if you don't need offload, avoiding this code path entirely
3. If adding a new backend, implement its branch in load_to_device_per_layer (and backup_from_device_all_layer)
4. Check SGLang version docs for the current list of supported io_backend values

Example fix

# before
server_args.io_backend = "cuda"  # unsupported
# after
server_args.io_backend = "kernel"

SUPPORTED_IO_BACKENDS = {"kernel", "direct", "kernel_ascend"}
assert server_args.io_backend in SUPPORTED_IO_BACKENDS, server_args.io_backend

def is_supported_io_backend(b: str) -> bool:
    return b in {"kernel", "direct", "kernel_ascend"}

try:
    host_pool.load_to_device_per_layer(...)
except ValueError as e:
    if "IO backend" in str(e):
        logger.error("Bad io_backend for hierarchical cache: %s", io_backend)
        raise

When it happens

Trigger: Calling load_to_device_per_layer with io_backend not in the handled set — e.g. 'cuda', 'nvshmem', an empty string, or a custom backend name not wired into this method.

Common situations: Setting --io-backend to an unsupported or misspelled value while hierarchical cache is enabled; new platforms whose backend name isn't registered here; forks adding an io_backend without implementing H2D transfer.

Related errors

• backend must be a non-empty string
• Cannot use the fast tokenizer in slow tokenizer mode.
• req_to_token table is empty but gather mask is non-empty.
• --enable-int8-mamba-checkpoint is not supported together…
• get_split_heads_page_buffer_meta requires…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6098b8d7f37709b4. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/mem_cache/pool_host/mha.py:370

                raise ValueError(f"Unsupported layout: {self.layout}")
        elif io_backend == "kernel_ascend":
            if self.layout == "page_first_direct":
                # Ascend-specific: transfer KV data for all layers when layer_id == 0
                if host_layer_id == 0:
                    transfer_kv_dim_exchange(
                        device_indices=device_indices,
                        host_indices=host_indices,
                        device_k=device_pool.k_buffer,
                        host_k=self.k_buffer,
                        device_v=device_pool.v_buffer,
                        host_v=self.v_buffer,
                        page_size=self.page_size,
                        direction=TransferDirection.H2D,
                    )
            else:
                raise ValueError(f"Unsupported layout: {self.layout}")
        else:
            raise ValueError(f"Unsupported IO backend: {io_backend}")

    def _resolve_device_transfer_buffers(self, device_pool):
        if self.mtp_draft_device_pools:
            return (
                self.packed_device_k_data_ptrs,
                self.packed_device_v_data_ptrs,
                self.packed_device_k_buffers,
                self.packed_device_v_buffers,
            )
        return (
            device_pool.k_data_ptrs,
            device_pool.v_data_ptrs,
            device_pool.k_buffer,
            device_pool.v_buffer,
        )

    def backup_from_device_all_layer(
        self, device_pool, host_indices, device_indices, io_backend

View on GitHub (pinned to 0132848349)