sgl-project/sglang · error · RuntimeError

KV4 is not tested on non-CUDA platforms.

Error message

KV4 is not tested on non-CUDA platforms.

What it means

The KV4 (FP4) KV cache path has only been validated on CUDA platforms. The _handle_kv4_compatibility resolution step's else-branch raises RuntimeError on non-CUDA builds (ROCm/HIP, XPU, CPU) rather than risk silently incorrect behavior.

Source

Thrown at python/sglang/srt/server_args.py:6715

                            "trtllm_mla",
                        ]
                        assert attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES, (
                            f"KV4 MLA expects attention_backend to be one of "
                            f"{KV4_ATTENTION_MLA_BACKEND_CHOICES}, but got {attention_backend}"
                        )
                    else:  # !FA4 + MHA
                        KV4_ATTENTION_MHA_BACKEND_CHOICES = [
                            "triton",
                            "torch_native",
                            "flex_attention",
                            "trtllm_mha",
                        ]
                        assert attention_backend in KV4_ATTENTION_MHA_BACKEND_CHOICES, (
                            f"KV4 MHA expects attention_backend to be one of "
                            f"{KV4_ATTENTION_MHA_BACKEND_CHOICES}, but got {attention_backend}"
                        )
        else:
            raise RuntimeError("KV4 is not tested on non-CUDA platforms.")

    def _handle_page_size(self):
        # Moved to the resolution pipeline (arg_groups/overrides.py:
        # _page_size_default), invoked here at its legacy slot.
        from sglang.srt.arg_groups.overrides import (
            _page_size_default,
            run_post_process_pass,
        )

        run_post_process_pass(self, _page_size_default)

    def _handle_amd_specifics(self):
        if is_hip():
            self._declare("_handle_amd_specifics", triton_attention_num_kv_splits=16)

    def _handle_nccl_pre_warm(self):
        # pre_warm_nccl is only used with CUDA or HIP hardware or NPU hardware
        cfg = resolving_view(self)

View on GitHub (pinned to 0132848349)

Solutions

1. Remove the FP4/KV4 kv-cache-dtype setting on non-CUDA platforms and use a supported dtype (fp8/bf16)
2. Run the KV4 path only on CUDA GPUs where it is tested
3. Track upstream SGLang for KV4 support on your platform

Example fix

# before (on ROCm/XPU)
python -m sglang.launch_server --model M --kv-cache-dtype nvfp4
# after
python -m sglang.launch_server --model M --kv-cache-dtype fp8_e4m3

import torch
if not torch.version.cuda:  # non-CUDA build (ROCm/XPU/CPU)
    assert not str(getattr(args, "kv_cache_dtype", "auto")).startswith("fp4"), (
        "KV4/FP4 KV cache is CUDA-only; use fp8/bf16")

try:
    ServerArgs(**kwargs)
except RuntimeError as e:
    if "KV4 is not tested on non-CUDA" in str(e):
        kwargs["kv_cache_dtype"] = "auto"
        ServerArgs(**kwargs)
    else:
        raise

When it happens

Trigger: is_cuda() is false while the KV4 compatibility handler runs — i.e. attempting to enable any KV4/FP4 kv_cache_dtype configuration on ROCm, Intel XPU, or CPU platforms.

Common situations: Running MI300x/ROCm or Intel GPU builds with configs copied from CUDA deployments that set an FP4 --kv-cache-dtype; enabling KV4 experimentally on non-CUDA backends where it was never tested.

Related errors

• Decode context parallel (decode_context_parallel_size > 1) i
• HIP does not support fused_marlin_moe currently.
• Only CUDA and MUSA support GGUF quantization currently.
• HiSparse supports DSA {label} backend(s) {sorted(allowed_bac
• HiSparse requires one of {HISPARSE_KV_CACHE_DTYPES} KV cache

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7ab0c617cfb29bcb. Report an issue: GitHub.