sgl-project/sglang · error · ValueError

{name} must be a torch.Tensor

Error message

{name} must be a torch.Tensor

What it means

sparse_mla_q8kv8_prefill_fwd validates q_scale and kv_scale in a loop; both must be torch.Tensor objects (not Python floats) because the kernel dereferences them as GPU buffers holding the quantization de-scale factors.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:423

    if attn_sink is not None:
        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:
            raise ValueError(
                f"attn_sink must be float32 with shape ({h_q},), got "
                f"{tuple(attn_sink.shape)}/{attn_sink.dtype}"
            )
        if not attn_sink.is_cuda:
            raise ValueError("attn_sink must be a CUDA tensor")
        if attn_sink.device != device:
            raise ValueError(
                f"attn_sink must be on q's device {device}, got {attn_sink.device}"
            )
        if not attn_sink.is_contiguous():
            raise ValueError("attn_sink must be contiguous")

    for name, scale in (("q_scale", q_scale), ("kv_scale", kv_scale)):
        if not isinstance(scale, torch.Tensor):
            raise ValueError(f"{name} must be a torch.Tensor")
        if not scale.is_cuda:
            raise ValueError(f"{name} must be a CUDA tensor")
        if scale.device != device:
            raise ValueError(
                f"{name} must be on q's device {device}, got {scale.device}"
            )
        if scale.dtype != torch.float32:
            raise ValueError(f"{name} must be float32, got {scale.dtype}")
        if scale.numel() != 1:
            raise ValueError(
                f"{name} must be a scalar tensor, got shape {tuple(scale.shape)}"
            )
        if not scale.is_contiguous():
            raise ValueError(f"{name} must be contiguous")

    if out is None:
        out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=device)
    else:

View on GitHub (pinned to 0132848349)

Solutions

  1. Wrap scalars: q_scale = torch.tensor(value, dtype=torch.float32, device=q.device)
  2. If scales come from a quantization config, precompute them as 1-element CUDA tensors once at setup

Example fix

// before
sparse_mla_q8kv8_prefill_fwd(q, k, v, ..., q_scale=1.0, kv_scale=0.5)
// after
dev = q.device
q_scale = torch.tensor(1.0, dtype=torch.float32, device=dev)
kv_scale = torch.tensor(0.5, dtype=torch.float32, device=dev)
sparse_mla_q8kv8_prefill_fwd(q, k, v, ..., q_scale=q_scale, kv_scale=kv_scale)
Defensive patterns

Strategy: type-guard

Validate before calling

if not isinstance(q_scale, torch.Tensor):
    q_scale = torch.tensor(q_scale, dtype=torch.float32, device=q.device)

Type guard

def as_scale_tensor(s, device) -> torch.Tensor:
    if not isinstance(s, torch.Tensor):
        s = torch.tensor(s, dtype=torch.float32, device=device)
    return s

Prevention

When it happens

Trigger: Calling the function with q_scale=1.0 or kv_scale=np.float32(0.5) — any non-tensor scalar (int, float, numpy scalar) triggers it.

Common situations: Porting code from an API that accepted float scales, or passing raw fp8 de-scale amplitudes computed in Python instead of materialized tensors.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/317d43312fd6608e. Report an issue: GitHub.