sgl-project/sglang · error · ValueError

Unsupported output_s dtype {output_s.dtype}

Error message

Unsupported output_s dtype {output_s.dtype}

What it means

The per-token-group quant kernel only supports two scale buffer dtypes: torch.int32 (UE8M0-packed) and torch.float32. Any other dtype (fp16, bf16, uint8, int64...) has no kernel implementation, so layout inference fails before launch.

Source

Thrown at python/sglang/kernels/ops/quantization/per_token_group_quant.py:82

def _infer_scale_layout(
    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int
) -> Tuple[bool, bool]:
    """Return ``(row_major, aligned)`` for ``output_s``.

    Column-major (transposed) scale buffers have token stride 1 and a larger
    group stride; row-major buffers are contiguous.
    """
    row_major = output_s.stride(-2) >= output_s.stride(-1)
    if output_s.dtype == torch.int32:
        if not scale_ue8m0:
            raise ValueError("int32-packed scale buffers require scale_ue8m0=True")
        aligned = num_groups % 4 == 0
        return row_major, aligned
    if output_s.dtype == torch.float32:
        if scale_ue8m0:
            raise ValueError("scale_ue8m0=True requires an int32-packed output_s")
        return row_major, True
    raise ValueError(f"Unsupported output_s dtype {output_s.dtype}")


@register_custom_op(
    op_name="per_token_group_quant",
    mutates_args=["output_q", "output_s"],
)
def _per_token_group_quant_custom_op(
    input: torch.Tensor,
    output_q: torch.Tensor,
    output_s: torch.Tensor,
    group_size: int,
    scale_ue8m0: bool = False,
    fuse_silu_and_mul: bool = False,
    masked_m: Optional[torch.Tensor] = None,
    expected_m: Optional[int] = None,
) -> None:
    num_groups = output_q.shape[-1] // group_size
    row_major, aligned = _infer_scale_layout(output_s, scale_ue8m0, num_groups)

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate output_s as torch.float32 for standard scales
2. Or torch.int32 with scale_ue8m0=True for packed UE8M0 scales

Example fix

# before
s = torch.empty(shape, dtype=torch.half, device='cuda')
# after
s = torch.empty(shape, dtype=torch.float32, device='cuda')

assert output_s.dtype in (torch.float32, torch.int32), output_s.dtype

def is_supported_scale_dtype(s): return s.dtype in (torch.float32, torch.int32)

When it happens

Trigger: Passing an output_s buffer with dtype other than int32/float32, e.g. torch.half or torch.uint8 allocated by caller code.

Common situations: Reusing a generic half-precision scratch buffer for scales, or downstream code that changed the scale tensor dtype between versions.

Related errors

• int32-packed scale buffers require scale_ue8m0=True
• scale_ue8m0=True requires an int32-packed output_s
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}
• {name} must have dtype {dtype}, got {t.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d88716b07b0cf3b3. Report an issue: GitHub.