{"record":{"id":"402df144293dfbb6","repo":"sgl-project/sglang","slug":"int32-packed-scale-buffers-require-scale-ue8m0-tru","errorCode":null,"errorMessage":"int32-packed scale buffers require scale_ue8m0=True","messagePattern":"int32-packed scale buffers require scale_ue8m0=True","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/quantization/per_token_group_quant.py","lineNumber":75,"sourceCode":"        \"masked\" if masked_layout else \"flat\",\n        cuda_files=[\"gemm/per_token_group_quant.cuh\"],\n        cuda_wrappers=[(\"per_token_group_quant\", f\"{launcher}<{trait_args}>::run\")],\n        extra_cuda_cflags=[\"--use_fast_math\"],\n    )\n\n\ndef _infer_scale_layout(\n    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int\n) -> Tuple[bool, bool]:\n    \"\"\"Return ``(row_major, aligned)`` for ``output_s``.\n\n    Column-major (transposed) scale buffers have token stride 1 and a larger\n    group stride; row-major buffers are contiguous.\n    \"\"\"\n    row_major = output_s.stride(-2) >= output_s.stride(-1)\n    if output_s.dtype == torch.int32:\n        if not scale_ue8m0:\n            raise ValueError(\"int32-packed scale buffers require scale_ue8m0=True\")\n        aligned = num_groups % 4 == 0\n        return row_major, aligned\n    if output_s.dtype == torch.float32:\n        if scale_ue8m0:\n            raise ValueError(\"scale_ue8m0=True requires an int32-packed output_s\")\n        return row_major, True\n    raise ValueError(f\"Unsupported output_s dtype {output_s.dtype}\")\n\n\n@register_custom_op(\n    op_name=\"per_token_group_quant\",\n    mutates_args=[\"output_q\", \"output_s\"],\n)\ndef _per_token_group_quant_custom_op(\n    input: torch.Tensor,\n    output_q: torch.Tensor,\n    output_s: torch.Tensor,\n    group_size: int,","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/quantization/per_token_group_quant.py#L57-L93","documentation":"_infer_scale_layout inspects the output scale buffer: an int32 scale tensor means scales are packed as four UE8M0 bytes per int32. That packing is only defined when scale_ue8m0=True; an int32 buffer with scale_ue8m0=False describes no valid encoding, so it errors.","triggerScenarios":"Calling per_token_group_quant with output_s of dtype torch.int32 but scale_ue8m0=False (default).","commonSituations":"Pre-allocating UE8M0 scale buffers (SM100-style, e.g. for cutlass NVFP4 kernels that want packed scales) while forgetting to set scale_ue8m0=True in the quant call; version upgrades that changed the scale dtype convention.","solutions":["Pass scale_ue8m0=True when output_s is int32 (UE8M0 packing)","Or allocate output_s as float32 if you want standard FP32 scales with scale_ue8m0=False"],"exampleFix":"# before\ns = torch.empty(..., dtype=torch.int32, device='cuda')\nper_token_group_quant(x, q, s)\n# after\ns = torch.empty(..., dtype=torch.int32, device='cuda')\nper_token_group_quant(x, q, s, scale_ue8m0=True)","handlingStrategy":"validation","validationCode":"if output_s.dtype == torch.int32:\n    assert scale_ue8m0, 'int32 scale buffer requires scale_ue8m0=True'","typeGuard":"def scale_config_valid(s, ue8m0): return (s.dtype == torch.int32) == ue8m0 or s.dtype == torch.float32 and not ue8m0","tryCatchPattern":null,"preventionTips":["Pair scale-buffer dtype with the ue8m0 flag in one helper","Add a unit test covering both scale modes"],"tags":["quantization","fp8","ue8m0","scale-factor","dtype-mismatch"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}