{"record":{"id":"4be41c5378b1fcc4","repo":"sgl-project/sglang","slug":"name-must-be-float32-got-scale-dtype","errorCode":null,"errorMessage":"{name} must be float32, got {scale.dtype}","messagePattern":"(.+?) must be float32, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":431,"sourceCode":"            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:\n            raise ValueError(\n                f\"{name} must be on q's device {device}, got {scale.device}\"\n            )\n        if scale.dtype != torch.float32:\n            raise ValueError(f\"{name} must be float32, got {scale.dtype}\")\n        if scale.numel() != 1:\n            raise ValueError(\n                f\"{name} must be a scalar tensor, got shape {tuple(scale.shape)}\"\n            )\n        if not scale.is_contiguous():\n            raise ValueError(f\"{name} must be contiguous\")\n\n    if out is None:\n        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)\n    else:\n        _check_out_buffer(out, \"out\", (s_q, h_q, d_v), torch.bfloat16, device)\n\n    if max_logits is None:\n        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)\n    else:\n        _check_out_buffer(max_logits, \"max_logits\", (s_q, h_q), torch.float32, device)\n\n    if lse is None:","sourceCodeStart":413,"sourceCodeEnd":449,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L413-L449","documentation":"q_scale and kv_scale must be float32 tensors. The kernel de-quantizes int8 q/k values with these scales in fp32 arithmetic; bf16/fp16/fp64 scales are rejected to avoid precision loss and implicit Triton type mismatches.","triggerScenarios":"Passing scales stored in bfloat16 (common when they ride along with bf16 model weights) or float64 (numpy defaults) as q_scale or kv_scale.","commonSituations":"Loading scales from a quantized checkpoint that stores everything in bf16, or converting from numpy (float64) via torch.from_numpy without a dtype cast.","solutions":["Cast at call site: q_scale = q_scale.float()","Store de-scale factors as float32 buffers when preparing the quantized model"],"exampleFix":"// before\nsparse_mla_q8kv8_prefill_fwd(..., q_scale=q_scale_bf16, kv_scale=kv_scale_bf16)\n// after\nsparse_mla_q8kv8_prefill_fwd(..., q_scale=q_scale_bf16.float(), kv_scale=kv_scale_bf16.float())","handlingStrategy":"validation","validationCode":"q_scale = q_scale.float() if q_scale.dtype != torch.float32 else q_scale\nkv_scale = kv_scale.float() if kv_scale.dtype != torch.float32 else kv_scale","typeGuard":"def fp32_scale(t: torch.Tensor) -> torch.Tensor:\n    return t if t.dtype == torch.float32 else t.float()","tryCatchPattern":null,"preventionTips":["Store fp8 de-scale factors in float32, not the weight dtype","Cast numpy-derived scales: torch.tensor(x, dtype=torch.float32)"],"tags":["mla","quantization","dtype","scale-validation"],"backgroundTag":"wrong-tensor-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}