{"record":{"id":"69e2ac913b2dc9b6","repo":"sgl-project/sglang","slug":"layer-sharded-hicache-backup-does-not-support-io-b","errorCode":null,"errorMessage":"Layer-sharded HiCache backup does not support IO backend: {io_backend}","messagePattern":"Layer-sharded HiCache backup does not support IO backend: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/mem_cache/pool_host/mla.py","lineNumber":407,"sourceCode":"                raise ValueError(\n                    f\"Layer-sharded HiCache backup does not support layout: {self.layout}\"\n                )\n        elif io_backend == \"direct\":\n            if self.layout == \"layer_first\":\n                transfer_kv_direct(\n                    src_layers=[device_pool.kv_buffer[device_layer_id]],\n                    dst_layers=[self.kv_buffer[host_layer_id]],\n                    src_indices=device_indices,\n                    dst_indices=host_indices,\n                    page_size=self.page_size,\n                )\n            else:\n                raise ValueError(\n                    \"Layer-sharded direct HiCache backup only supports \"\n                    f\"layer_first layout, got {self.layout}\"\n                )\n        else:\n            raise ValueError(\n                f\"Layer-sharded HiCache backup does not support IO backend: {io_backend}\"\n            )\n\n    def _resolve_device_transfer_buffers(self, device_pool):\n        if self.mtp_draft_device_pools:\n            return self.packed_device_data_ptrs, self.packed_device_kv_buffers\n        return device_pool.data_ptrs, device_pool.kv_buffer\n\n    def backup_from_device_all_layer(\n        self, device_pool, host_indices, device_indices, io_backend\n    ):\n        host_indices = self.maybe_dcp_kernel_indices(host_indices)\n        device_indices = self.maybe_dcp_kernel_indices(device_indices)\n        if self._is_device_layer_sharded(device_pool):\n            for layer_id in self._owned_device_layer_ids(device_pool):\n                self._backup_from_device_per_layer(\n                    device_pool, host_indices, device_indices, layer_id, io_backend\n                )","sourceCodeStart":389,"sourceCodeEnd":425,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/pool_host/mla.py#L389-L425","documentation":"_backup_from_device_per_layer accepts only io_backend values \"kernel\" and \"direct\". Any other value (e.g. \"kernel_ascend\", \"vanilla\", a typo, or None) reaches the final else and raises, because no per-layer implementation exists for that IO backend.","triggerScenarios":"backup_from_device_all_layer receives an io_backend string outside {kernel, direct} while the device pool is layer-sharded, so the per-layer loop is taken and the backend dispatch fails.","commonSituations":"Setting --hicache-io-backend kernel_ascend (or an unset/misspelled value) on a CP-layer-sharded deployment; a new IO backend added to backup_from_device_all_layer without a per-layer counterpart.","solutions":["Use io_backend=\"kernel\" or \"direct\" for layer-sharded configurations","Fix or validate the --hicache-io-backend server argument before startup","Disable layer sharding so the all-layer dispatch (which supports kernel_ascend) is used"],"exampleFix":"# before\nserver_args.hicache_io_backend = \"kernel_ascend\"  # with CP layer sharding\n# after\nserver_args.hicache_io_backend = \"kernel\"","handlingStrategy":"validation","validationCode":"VALID_BACKENDS = {\"kernel\", \"direct\"}\nif io_backend not in VALID_BACKENDS:\n    raise ConfigError(f\"io_backend must be one of {VALID_BACKENDS}, got {io_backend!r}\")","typeGuard":"def is_valid_per_layer_backend(b: str) -> bool:\n    return b in (\"kernel\", \"direct\")","tryCatchPattern":"try:\n    pool.backup_from_device_all_layer(dp, hi, di, io_backend)\nexcept ValueError as e:\n    if \"does not support IO backend\" in str(e):\n        io_backend = \"kernel\"\n        pool.backup_from_device_all_layer(dp, hi, di, io_backend)\n    else:\n        raise","preventionTips":["Validate --hicache-io-backend against the allowed enum at arg parsing","Fail fast on typos and case mismatches before training/serving starts"],"tags":["hicache","io-backend","context-parallelism","sglang"],"backgroundTag":"unsupported-configuration-combination","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}