sgl-project/sglang · error · ValueError

scale_ue8m0=True requires an int32-packed output_s

Error message

scale_ue8m0=True requires an int32-packed output_s

What it means

The scale_ue8m0=True option means scales are UE8M0 exponent bytes packed four-per-int32. Therefore it requires the output scale buffer to be torch.int32; a float32 buffer cannot hold the packed representation, so the layout inference rejects the combination.

Source

Thrown at python/sglang/kernels/ops/quantization/per_token_group_quant.py:80

def _infer_scale_layout(
    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int
) -> Tuple[bool, bool]:
    """Return ``(row_major, aligned)`` for ``output_s``.

    Column-major (transposed) scale buffers have token stride 1 and a larger
    group stride; row-major buffers are contiguous.
    """
    row_major = output_s.stride(-2) >= output_s.stride(-1)
    if output_s.dtype == torch.int32:
        if not scale_ue8m0:
            raise ValueError("int32-packed scale buffers require scale_ue8m0=True")
        aligned = num_groups % 4 == 0
        return row_major, aligned
    if output_s.dtype == torch.float32:
        if scale_ue8m0:
            raise ValueError("scale_ue8m0=True requires an int32-packed output_s")
        return row_major, True
    raise ValueError(f"Unsupported output_s dtype {output_s.dtype}")


@register_custom_op(
    op_name="per_token_group_quant",
    mutates_args=["output_q", "output_s"],
)
def _per_token_group_quant_custom_op(
    input: torch.Tensor,
    output_q: torch.Tensor,
    output_s: torch.Tensor,
    group_size: int,
    scale_ue8m0: bool = False,
    fuse_silu_and_mul: bool = False,
    masked_m: Optional[torch.Tensor] = None,
    expected_m: Optional[int] = None,
) -> None:

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate output_s as torch.int32 when using scale_ue8m0=True
2. Or drop scale_ue8m0 if the consumer kernel expects float32 scales

Example fix

# before
s = torch.empty(..., dtype=torch.float32, device='cuda')
per_token_group_quant(x, q, s, scale_ue8m0=True)
# after
s = torch.empty((..., snum//4), dtype=torch.int32, device='cuda')
per_token_group_quant(x, q, s, scale_ue8m0=True)

if scale_ue8m0:
    assert output_s.dtype == torch.int32

def ue8m0_buffer_ok(s): return s.dtype == torch.int32

When it happens

Trigger: Calling per_token_group_quant with scale_ue8m0=True but output_s.dtype == torch.float32.

Common situations: Enabling UE8M0 (Blackwell-oriented) scales while reusing an old float32 scale allocation; copy-pasted buffer allocation from a non-UE8M0 path.

Related errors

• int32-packed scale buffers require scale_ue8m0=True
• Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}
• Unsupported output_s dtype {output_s.dtype}
• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2fd01066d3bec3a9. Report an issue: GitHub.