sgl-project/sglang · error · ValueError

{name} must be a scalar tensor, got shape {tuple(scale.shape

Error message

{name} must be a scalar tensor, got shape {tuple(scale.shape)}

What it means

q_scale and kv_scale must each be a 1-element (scalar-shaped) tensor, e.g. shape () or (1,). The kernel loads a single scale value; multi-element scales would leave the de-scale factor ambiguous.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:433

            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(
                f"{name} must be a scalar tensor, got shape {tuple(scale.shape)}"
            )
        if not scale.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if out is None:
        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)
    else:
        _check_out_buffer(out, "out", (s_q, h_q, d_v), torch.bfloat16, device)

    if max_logits is None:
        max_logits = torch.empty(s_q, h_q, dtype=torch.float32, device=device)
    else:
        _check_out_buffer(max_logits, "max_logits", (s_q, h_q), torch.float32, device)

    if lse is None:
        lse = torch.empty(s_q, h_q, dtype=torch.float32, device=device)
    else:

View on GitHub (pinned to 0132848349)

Solutions

Reduce to a scalar: q_scale = q_scale_vector.mean().float().reshape(1) if a single value is acceptable
Use the correct per-tensor de-scale values from the quantization recipe rather than per-channel scales

Example fix

// before
q_scale = per_channel_scales  # shape (h_q,)
// after
q_scale = torch.tensor(calibrated_per_tensor_scale, dtype=torch.float32, device=q.device)

Defensive patterns

Strategy: validation

Validate before calling

assert q_scale.numel() == 1 and kv_scale.numel() == 1

Type guard

def is_scalar_tensor(t: torch.Tensor) -> bool:
    return isinstance(t, torch.Tensor) and t.numel() == 1

Prevention

Confirm the quantization recipe is per-tensor before using this kernel
Reduce per-channel scales to a single calibrated value at setup

When it happens

Trigger: Passing per-head or per-group scale vectors with shape (h_q,) or (num_groups,) as q_scale/kv_scale — this kernel only supports scalar (per-tensor) quantization scales.

Common situations: Migrating from a per-channel quantized attention backend whose scale tensors are vectors, then handing them to this per-tensor-scaled kernel unchanged.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/132925b4fd21b9d9. Report an issue: GitHub.