{"record":{"id":"5e6b7098a4ef785f","repo":"sgl-project/sglang","slug":"unsupported-dtype-k-dtype-supported-bfloat16","errorCode":null,"errorMessage":"Unsupported dtype {k.dtype}. Supported: bfloat16, float16","messagePattern":"Unsupported dtype (.+?)\\. Supported: bfloat16, float16","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/kvcache/fused_fp8_qkv_kv_cache.py","lineNumber":47,"sourceCode":"def _scale_to_f32(scale: Optional[torch.Tensor], device: torch.device) -> torch.Tensor:\n    if scale is None:\n        return torch.ones(1, dtype=torch.float32, device=device)\n    return scale.to(torch.float32).reshape(1)\n\n\ndef fused_fp8_qkv_kv_cache(\n    q: torch.Tensor | None,\n    k: torch.Tensor,\n    v: torch.Tensor,\n    k_cache: torch.Tensor,\n    v_cache: torch.Tensor,\n    cache_loc: torch.Tensor,\n    k_scale: Optional[torch.Tensor] = None,\n    v_scale: Optional[torch.Tensor] = None,\n) -> torch.Tensor | None:\n    \"\"\"Fused FP8 quant of K/V (+ optional Q) + paged KV-cache write.\"\"\"\n    if k.dtype not in (torch.bfloat16, torch.float16):\n        raise RuntimeError(f\"Unsupported dtype {k.dtype}. Supported: bfloat16, float16\")\n\n    num_tokens = k.shape[0]\n    k2 = k.reshape(num_tokens, -1)\n    v2 = v.reshape(num_tokens, -1)\n    kv_dim = k2.shape[1]\n\n    k_cache2 = k_cache.view(-1, kv_dim)\n    v_cache2 = v_cache.view(-1, kv_dim)\n\n    ks = _scale_to_f32(k_scale, k.device)\n    vs = _scale_to_f32(v_scale, k.device)\n\n    q2 = None\n    q_out = None\n    if q is not None:\n        q2 = q.reshape(num_tokens, -1)\n        q_out = torch.empty(q2.shape, dtype=torch.float8_e4m3fn, device=q.device)\n","sourceCodeStart":29,"sourceCodeEnd":65,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/kvcache/fused_fp8_qkv_kv_cache.py#L29-L65","documentation":"fused_fp8_qkv_kv_cache quantizes K/V (and optionally Q) to FP8 and writes into the paged KV cache; quantization kernels only accept BF16/FP16 inputs. Other dtypes (FP32, FP8 input, etc.) have no quantization path, so the op raises RuntimeError.","triggerScenarios":"Calling fused_fp8_qkv_kv_cache(k, v, ...) where k.dtype is torch.float32 (or anything other than bfloat16/float16), e.g. after a model or projection layer kept activations in FP32.","commonSituations":"Running with --dtype float32; a debug/reference path casting projections to FP32; upcast side effects from autocast disabled; passing pre-quantized FP8 tensors meant for a different kernel.","solutions":["Cast K and V to bfloat16 (or float16) before calling: k.to(torch.bfloat16)","Launch the server/model with bf16/fp16 dtype instead of float32","If inputs are already FP8, use the FP8-native cache write op instead of this quantization wrapper"],"exampleFix":"# before\nout = fused_fp8_qkv_kv_cache(k, v, q, cache_loc, ...)\n# after\nout = fused_fp8_qkv_kv_cache(k.to(torch.bfloat16), v.to(torch.bfloat16),\n                             q.to(torch.bfloat16) if q is not None else None, cache_loc, ...)","handlingStrategy":"type-guard","validationCode":"assert k.dtype in (torch.bfloat16, torch.float16), f\"got {k.dtype}\"","typeGuard":"def kv_dtype_supported(k: torch.Tensor, v: torch.Tensor) -> bool:\n    return k.dtype in (torch.bfloat16, torch.float16) and v.dtype in (torch.bfloat16, torch.float16)","tryCatchPattern":null,"preventionTips":["Run models with bf16/fp16 dtype","Cast projections to bfloat16 at the model boundary instead of relying on downstream casts"],"tags":["fp8","kv-cache","dtype-validation"],"backgroundTag":"unsupported-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}