{"record":{"id":"2fd01066d3bec3a9","repo":"sgl-project/sglang","slug":"scale-ue8m0-true-requires-an-int32-packed-output-s","errorCode":null,"errorMessage":"scale_ue8m0=True requires an int32-packed output_s","messagePattern":"scale_ue8m0=True requires an int32-packed output_s","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/quantization/per_token_group_quant.py","lineNumber":80,"sourceCode":"\n\ndef _infer_scale_layout(\n    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int\n) -> Tuple[bool, bool]:\n    \"\"\"Return ``(row_major, aligned)`` for ``output_s``.\n\n    Column-major (transposed) scale buffers have token stride 1 and a larger\n    group stride; row-major buffers are contiguous.\n    \"\"\"\n    row_major = output_s.stride(-2) >= output_s.stride(-1)\n    if output_s.dtype == torch.int32:\n        if not scale_ue8m0:\n            raise ValueError(\"int32-packed scale buffers require scale_ue8m0=True\")\n        aligned = num_groups % 4 == 0\n        return row_major, aligned\n    if output_s.dtype == torch.float32:\n        if scale_ue8m0:\n            raise ValueError(\"scale_ue8m0=True requires an int32-packed output_s\")\n        return row_major, True\n    raise ValueError(f\"Unsupported output_s dtype {output_s.dtype}\")\n\n\n@register_custom_op(\n    op_name=\"per_token_group_quant\",\n    mutates_args=[\"output_q\", \"output_s\"],\n)\ndef _per_token_group_quant_custom_op(\n    input: torch.Tensor,\n    output_q: torch.Tensor,\n    output_s: torch.Tensor,\n    group_size: int,\n    scale_ue8m0: bool = False,\n    fuse_silu_and_mul: bool = False,\n    masked_m: Optional[torch.Tensor] = None,\n    expected_m: Optional[int] = None,\n) -> None:","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/quantization/per_token_group_quant.py#L62-L98","documentation":"The scale_ue8m0=True option means scales are UE8M0 exponent bytes packed four-per-int32. Therefore it requires the output scale buffer to be torch.int32; a float32 buffer cannot hold the packed representation, so the layout inference rejects the combination.","triggerScenarios":"Calling per_token_group_quant with scale_ue8m0=True but output_s.dtype == torch.float32.","commonSituations":"Enabling UE8M0 (Blackwell-oriented) scales while reusing an old float32 scale allocation; copy-pasted buffer allocation from a non-UE8M0 path.","solutions":["Allocate output_s as torch.int32 when using scale_ue8m0=True","Or drop scale_ue8m0 if the consumer kernel expects float32 scales"],"exampleFix":"# before\ns = torch.empty(..., dtype=torch.float32, device='cuda')\nper_token_group_quant(x, q, s, scale_ue8m0=True)\n# after\ns = torch.empty((..., snum//4), dtype=torch.int32, device='cuda')\nper_token_group_quant(x, q, s, scale_ue8m0=True)","handlingStrategy":"validation","validationCode":"if scale_ue8m0:\n    assert output_s.dtype == torch.int32","typeGuard":"def ue8m0_buffer_ok(s): return s.dtype == torch.int32","tryCatchPattern":null,"preventionTips":["Centralize scale-buffer allocation next to the scale_ue8m0 decision"],"tags":["quantization","fp8","ue8m0","scale-factor","dtype-mismatch"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}