{"record":{"id":"7ab0c617cfb29bcb","repo":"sgl-project/sglang","slug":"kv4-is-not-tested-on-non-cuda-platforms","errorCode":null,"errorMessage":"KV4 is not tested on non-CUDA platforms.","messagePattern":"KV4 is not tested on non-CUDA platforms\\.","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/server_args.py","lineNumber":6715,"sourceCode":"                            \"trtllm_mla\",\n                        ]\n                        assert attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES, (\n                            f\"KV4 MLA expects attention_backend to be one of \"\n                            f\"{KV4_ATTENTION_MLA_BACKEND_CHOICES}, but got {attention_backend}\"\n                        )\n                    else:  # !FA4 + MHA\n                        KV4_ATTENTION_MHA_BACKEND_CHOICES = [\n                            \"triton\",\n                            \"torch_native\",\n                            \"flex_attention\",\n                            \"trtllm_mha\",\n                        ]\n                        assert attention_backend in KV4_ATTENTION_MHA_BACKEND_CHOICES, (\n                            f\"KV4 MHA expects attention_backend to be one of \"\n                            f\"{KV4_ATTENTION_MHA_BACKEND_CHOICES}, but got {attention_backend}\"\n                        )\n        else:\n            raise RuntimeError(\"KV4 is not tested on non-CUDA platforms.\")\n\n    def _handle_page_size(self):\n        # Moved to the resolution pipeline (arg_groups/overrides.py:\n        # _page_size_default), invoked here at its legacy slot.\n        from sglang.srt.arg_groups.overrides import (\n            _page_size_default,\n            run_post_process_pass,\n        )\n\n        run_post_process_pass(self, _page_size_default)\n\n    def _handle_amd_specifics(self):\n        if is_hip():\n            self._declare(\"_handle_amd_specifics\", triton_attention_num_kv_splits=16)\n\n    def _handle_nccl_pre_warm(self):\n        # pre_warm_nccl is only used with CUDA or HIP hardware or NPU hardware\n        cfg = resolving_view(self)","sourceCodeStart":6697,"sourceCodeEnd":6733,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/server_args.py#L6697-L6733","documentation":"The KV4 (FP4) KV cache path has only been validated on CUDA platforms. The _handle_kv4_compatibility resolution step's else-branch raises RuntimeError on non-CUDA builds (ROCm/HIP, XPU, CPU) rather than risk silently incorrect behavior.","triggerScenarios":"is_cuda() is false while the KV4 compatibility handler runs — i.e. attempting to enable any KV4/FP4 kv_cache_dtype configuration on ROCm, Intel XPU, or CPU platforms.","commonSituations":"Running MI300x/ROCm or Intel GPU builds with configs copied from CUDA deployments that set an FP4 --kv-cache-dtype; enabling KV4 experimentally on non-CUDA backends where it was never tested.","solutions":["Remove the FP4/KV4 kv-cache-dtype setting on non-CUDA platforms and use a supported dtype (fp8/bf16)","Run the KV4 path only on CUDA GPUs where it is tested","Track upstream SGLang for KV4 support on your platform"],"exampleFix":"# before (on ROCm/XPU)\npython -m sglang.launch_server --model M --kv-cache-dtype nvfp4\n# after\npython -m sglang.launch_server --model M --kv-cache-dtype fp8_e4m3","handlingStrategy":"validation","validationCode":"import torch\nif not torch.version.cuda:  # non-CUDA build (ROCm/XPU/CPU)\n    assert not str(getattr(args, \"kv_cache_dtype\", \"auto\")).startswith(\"fp4\"), (\n        \"KV4/FP4 KV cache is CUDA-only; use fp8/bf16\")","typeGuard":null,"tryCatchPattern":"try:\n    ServerArgs(**kwargs)\nexcept RuntimeError as e:\n    if \"KV4 is not tested on non-CUDA\" in str(e):\n        kwargs[\"kv_cache_dtype\"] = \"auto\"\n        ServerArgs(**kwargs)\n    else:\n        raise","preventionTips":["Gate FP4 KV-cache flags behind a torch.version.cuda check in launch templates","Keep separate config sets for CUDA and ROCm/XPU deployments","Watch release notes before enabling KV4 on non-CUDA backends"],"tags":["sglang","kv4","platform-support","rocm","non-cuda","kv-cache-dtype"],"backgroundTag":"platform-unsupported-feature","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}