sgl-project/sglang · error · ValueError

int32-packed scale buffers require scale_ue8m0=True

Error message

int32-packed scale buffers require scale_ue8m0=True

What it means

_infer_scale_layout inspects the output scale buffer: an int32 scale tensor means scales are packed as four UE8M0 bytes per int32. That packing is only defined when scale_ue8m0=True; an int32 buffer with scale_ue8m0=False describes no valid encoding, so it errors.

Source

Thrown at python/sglang/kernels/ops/quantization/per_token_group_quant.py:75

        "masked" if masked_layout else "flat",
        cuda_files=["gemm/per_token_group_quant.cuh"],
        cuda_wrappers=[("per_token_group_quant", f"{launcher}<{trait_args}>::run")],
        extra_cuda_cflags=["--use_fast_math"],
    )


def _infer_scale_layout(
    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int
) -> Tuple[bool, bool]:
    """Return ``(row_major, aligned)`` for ``output_s``.

    Column-major (transposed) scale buffers have token stride 1 and a larger
    group stride; row-major buffers are contiguous.
    """
    row_major = output_s.stride(-2) >= output_s.stride(-1)
    if output_s.dtype == torch.int32:
        if not scale_ue8m0:
            raise ValueError("int32-packed scale buffers require scale_ue8m0=True")
        aligned = num_groups % 4 == 0
        return row_major, aligned
    if output_s.dtype == torch.float32:
        if scale_ue8m0:
            raise ValueError("scale_ue8m0=True requires an int32-packed output_s")
        return row_major, True
    raise ValueError(f"Unsupported output_s dtype {output_s.dtype}")


@register_custom_op(
    op_name="per_token_group_quant",
    mutates_args=["output_q", "output_s"],
)
def _per_token_group_quant_custom_op(
    input: torch.Tensor,
    output_q: torch.Tensor,
    output_s: torch.Tensor,
    group_size: int,

View on GitHub (pinned to 0132848349)

Solutions

1. Pass scale_ue8m0=True when output_s is int32 (UE8M0 packing)
2. Or allocate output_s as float32 if you want standard FP32 scales with scale_ue8m0=False

Example fix

# before
s = torch.empty(..., dtype=torch.int32, device='cuda')
per_token_group_quant(x, q, s)
# after
s = torch.empty(..., dtype=torch.int32, device='cuda')
per_token_group_quant(x, q, s, scale_ue8m0=True)

if output_s.dtype == torch.int32:
    assert scale_ue8m0, 'int32 scale buffer requires scale_ue8m0=True'

def scale_config_valid(s, ue8m0): return (s.dtype == torch.int32) == ue8m0 or s.dtype == torch.float32 and not ue8m0

When it happens

Trigger: Calling per_token_group_quant with output_s of dtype torch.int32 but scale_ue8m0=False (default).

Common situations: Pre-allocating UE8M0 scale buffers (SM100-style, e.g. for cutlass NVFP4 kernels that want packed scales) while forgetting to set scale_ue8m0=True in the quant call; version upgrades that changed the scale dtype convention.

Related errors

• scale_ue8m0=True requires an int32-packed output_s
• Type mismatch: {self.sfq_dtype} != {self.sfk_dtype}
• Unsupported output_s dtype {output_s.dtype}
• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/402df144293dfbb6. Report an issue: GitHub.