sgl-project/sglang · error · ValueError

Layer-sharded direct HiCache backup only supports layer_firs

Error message

Layer-sharded direct HiCache backup only supports layer_first layout, got {self.layout}

What it means

When io_backend is "direct" and the device pool is layer-sharded, the per-layer backup only implements the layer_first host layout via transfer_kv_direct. Any other layout (page_first, page_first_kv_split, ...) hits this ValueError because a direct per-layer transfer would need scatter into a page-strided host buffer, which is not implemented.

Source

Thrown at python/sglang/srt/mem_cache/pool_host/mla.py:402

                    raise ValueError(
                        "Layer-sharded MLA HiCache backup with page_first layout "
                        "requires the JIT one-layer kernel."
                    )
            else:
                raise ValueError(
                    f"Layer-sharded HiCache backup does not support layout: {self.layout}"
                )
        elif io_backend == "direct":
            if self.layout == "layer_first":
                transfer_kv_direct(
                    src_layers=[device_pool.kv_buffer[device_layer_id]],
                    dst_layers=[self.kv_buffer[host_layer_id]],
                    src_indices=device_indices,
                    dst_indices=host_indices,
                    page_size=self.page_size,
                )
            else:
                raise ValueError(
                    "Layer-sharded direct HiCache backup only supports "
                    f"layer_first layout, got {self.layout}"
                )
        else:
            raise ValueError(
                f"Layer-sharded HiCache backup does not support IO backend: {io_backend}"
            )

    def _resolve_device_transfer_buffers(self, device_pool):
        if self.mtp_draft_device_pools:
            return self.packed_device_data_ptrs, self.packed_device_kv_buffers
        return device_pool.data_ptrs, device_pool.kv_buffer

    def backup_from_device_all_layer(
        self, device_pool, host_indices, device_indices, io_backend
    ):
        host_indices = self.maybe_dcp_kernel_indices(host_indices)
        device_indices = self.maybe_dcp_kernel_indices(device_indices)

View on GitHub (pinned to 0132848349)

Solutions

Use io_backend="kernel" instead, which supports both layer_first and page_first
Switch the host layout to layer_first so the direct per-layer path applies
Avoid layer-sharded device pools (no CP) when you must use direct IO with page-first layouts

Example fix

# before
server_args.hicache_io_backend = "direct"   # layer-sharded pool + page_first
# after
server_args.hicache_io_backend = "kernel"

Defensive patterns

Strategy: validation

Validate before calling

if io_backend == "direct" and sharded and pool.layout != "layer_first":
    io_backend = "kernel"  # direct per-layer needs layer_first

Type guard

def direct_per_layer_ok(pool) -> bool:
    return pool.layout == "layer_first"

Try / catch

try:
    pool.backup_from_device_all_layer(dp, hi, di, io_backend)
except ValueError as e:
    if "direct HiCache backup only supports" in str(e):
        io_backend = "kernel"
        pool.backup_from_device_all_layer(dp, hi, di, io_backend)
    else:
        raise

Prevention

Map direct IO to layer_first layout in configuration
Centralize backend/layout compatibility checks in one helper used by all transfer entry points

When it happens

Trigger: Calling backup_from_device_all_layer with io_backend="direct" on a layer-sharded device pool while the host pool layout is page_first or another non-layer_first layout.

Common situations: Setting --hicache-io-backend direct (NIXL/GDS path) on a multi-GPU CP run where the host buffer was allocated page-first; or enabling page_first_direct expecting the direct backend to work per-layer.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/735331ab048a6aea. Report an issue: GitHub.