sgl-project/sglang · error · RuntimeError

Unsupported dtype {k.dtype}. Supported: bfloat16, float16

Error message

Unsupported dtype {k.dtype}. Supported: bfloat16, float16

What it means

fused_fp8_qkv_kv_cache quantizes K/V (and optionally Q) to FP8 and writes into the paged KV cache; quantization kernels only accept BF16/FP16 inputs. Other dtypes (FP32, FP8 input, etc.) have no quantization path, so the op raises RuntimeError.

Source

Thrown at python/sglang/kernels/ops/kvcache/fused_fp8_qkv_kv_cache.py:47

def _scale_to_f32(scale: Optional[torch.Tensor], device: torch.device) -> torch.Tensor:
    if scale is None:
        return torch.ones(1, dtype=torch.float32, device=device)
    return scale.to(torch.float32).reshape(1)


def fused_fp8_qkv_kv_cache(
    q: torch.Tensor | None,
    k: torch.Tensor,
    v: torch.Tensor,
    k_cache: torch.Tensor,
    v_cache: torch.Tensor,
    cache_loc: torch.Tensor,
    k_scale: Optional[torch.Tensor] = None,
    v_scale: Optional[torch.Tensor] = None,
) -> torch.Tensor | None:
    """Fused FP8 quant of K/V (+ optional Q) + paged KV-cache write."""
    if k.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(f"Unsupported dtype {k.dtype}. Supported: bfloat16, float16")

    num_tokens = k.shape[0]
    k2 = k.reshape(num_tokens, -1)
    v2 = v.reshape(num_tokens, -1)
    kv_dim = k2.shape[1]

    k_cache2 = k_cache.view(-1, kv_dim)
    v_cache2 = v_cache.view(-1, kv_dim)

    ks = _scale_to_f32(k_scale, k.device)
    vs = _scale_to_f32(v_scale, k.device)

    q2 = None
    q_out = None
    if q is not None:
        q2 = q.reshape(num_tokens, -1)
        q_out = torch.empty(q2.shape, dtype=torch.float8_e4m3fn, device=q.device)

View on GitHub (pinned to 0132848349)

Solutions

1. Cast K and V to bfloat16 (or float16) before calling: k.to(torch.bfloat16)
2. Launch the server/model with bf16/fp16 dtype instead of float32
3. If inputs are already FP8, use the FP8-native cache write op instead of this quantization wrapper

Example fix

# before
out = fused_fp8_qkv_kv_cache(k, v, q, cache_loc, ...)
# after
out = fused_fp8_qkv_kv_cache(k.to(torch.bfloat16), v.to(torch.bfloat16),
                             q.to(torch.bfloat16) if q is not None else None, cache_loc, ...)

assert k.dtype in (torch.bfloat16, torch.float16), f"got {k.dtype}"

def kv_dtype_supported(k: torch.Tensor, v: torch.Tensor) -> bool:
    return k.dtype in (torch.bfloat16, torch.float16) and v.dtype in (torch.bfloat16, torch.float16)

When it happens

Trigger: Calling fused_fp8_qkv_kv_cache(k, v, ...) where k.dtype is torch.float32 (or anything other than bfloat16/float16), e.g. after a model or projection layer kept activations in FP32.

Common situations: Running with --dtype float32; a debug/reference path casting projections to FP32; upcast side effects from autocast disabled; passing pre-quantized FP8 tensors meant for a different kernel.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• kv_scales must be fp32, got {kv_scales.dtype}
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE
• kv_scales shape {tuple(kv_scales.shape)} does not match expe
• {name} must have dtype {dtype}, got {t.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5e6b7098a4ef785f. Report an issue: GitHub.