sgl-project/sglang · error · ValueError

{name} must be float32, got {scale.dtype}

Error message

{name} must be float32, got {scale.dtype}

What it means

q_scale and kv_scale must be float32 tensors. The kernel de-quantizes int8 q/k values with these scales in fp32 arithmetic; bf16/fp16/fp64 scales are rejected to avoid precision loss and implicit Triton type mismatches.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:431

            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(
                f"{name} must be a scalar tensor, got shape {tuple(scale.shape)}"
            )
        if not scale.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if out is None:
        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)
    else:
        _check_out_buffer(out, "out", (s_q, h_q, d_v), torch.bfloat16, device)

    if max_logits is None:
        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)
    else:
        _check_out_buffer(max_logits, "max_logits", (s_q, h_q), torch.float32, device)

    if lse is None:

View on GitHub (pinned to 0132848349)

Solutions

  1. Cast at call site: q_scale = q_scale.float()
  2. Store de-scale factors as float32 buffers when preparing the quantized model

Example fix

// before
sparse_mla_q8kv8_prefill_fwd(..., q_scale=q_scale_bf16, kv_scale=kv_scale_bf16)
// after
sparse_mla_q8kv8_prefill_fwd(..., q_scale=q_scale_bf16.float(), kv_scale=kv_scale_bf16.float())
Defensive patterns

Strategy: validation

Validate before calling

q_scale = q_scale.float() if q_scale.dtype != torch.float32 else q_scale
kv_scale = kv_scale.float() if kv_scale.dtype != torch.float32 else kv_scale

Type guard

def fp32_scale(t: torch.Tensor) -> torch.Tensor:
    return t if t.dtype == torch.float32 else t.float()

Prevention

When it happens

Trigger: Passing scales stored in bfloat16 (common when they ride along with bf16 model weights) or float64 (numpy defaults) as q_scale or kv_scale.

Common situations: Loading scales from a quantized checkpoint that stores everything in bf16, or converting from numpy (float64) via torch.from_numpy without a dtype cast.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4be41c5378b1fcc4. Report an issue: GitHub.