{"record":{"id":"d88716b07b0cf3b3","repo":"sgl-project/sglang","slug":"unsupported-output-s-dtype-output-s-dtype","errorCode":null,"errorMessage":"Unsupported output_s dtype {output_s.dtype}","messagePattern":"Unsupported output_s dtype (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/quantization/per_token_group_quant.py","lineNumber":82,"sourceCode":"def _infer_scale_layout(\n    output_s: torch.Tensor, scale_ue8m0: bool, num_groups: int\n) -> Tuple[bool, bool]:\n    \"\"\"Return ``(row_major, aligned)`` for ``output_s``.\n\n    Column-major (transposed) scale buffers have token stride 1 and a larger\n    group stride; row-major buffers are contiguous.\n    \"\"\"\n    row_major = output_s.stride(-2) >= output_s.stride(-1)\n    if output_s.dtype == torch.int32:\n        if not scale_ue8m0:\n            raise ValueError(\"int32-packed scale buffers require scale_ue8m0=True\")\n        aligned = num_groups % 4 == 0\n        return row_major, aligned\n    if output_s.dtype == torch.float32:\n        if scale_ue8m0:\n            raise ValueError(\"scale_ue8m0=True requires an int32-packed output_s\")\n        return row_major, True\n    raise ValueError(f\"Unsupported output_s dtype {output_s.dtype}\")\n\n\n@register_custom_op(\n    op_name=\"per_token_group_quant\",\n    mutates_args=[\"output_q\", \"output_s\"],\n)\ndef _per_token_group_quant_custom_op(\n    input: torch.Tensor,\n    output_q: torch.Tensor,\n    output_s: torch.Tensor,\n    group_size: int,\n    scale_ue8m0: bool = False,\n    fuse_silu_and_mul: bool = False,\n    masked_m: Optional[torch.Tensor] = None,\n    expected_m: Optional[int] = None,\n) -> None:\n    num_groups = output_q.shape[-1] // group_size\n    row_major, aligned = _infer_scale_layout(output_s, scale_ue8m0, num_groups)","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/quantization/per_token_group_quant.py#L64-L100","documentation":"The per-token-group quant kernel only supports two scale buffer dtypes: torch.int32 (UE8M0-packed) and torch.float32. Any other dtype (fp16, bf16, uint8, int64...) has no kernel implementation, so layout inference fails before launch.","triggerScenarios":"Passing an output_s buffer with dtype other than int32/float32, e.g. torch.half or torch.uint8 allocated by caller code.","commonSituations":"Reusing a generic half-precision scratch buffer for scales, or downstream code that changed the scale tensor dtype between versions.","solutions":["Allocate output_s as torch.float32 for standard scales","Or torch.int32 with scale_ue8m0=True for packed UE8M0 scales"],"exampleFix":"# before\ns = torch.empty(shape, dtype=torch.half, device='cuda')\n# after\ns = torch.empty(shape, dtype=torch.float32, device='cuda')","handlingStrategy":"validation","validationCode":"assert output_s.dtype in (torch.float32, torch.int32), output_s.dtype","typeGuard":"def is_supported_scale_dtype(s): return s.dtype in (torch.float32, torch.int32)","tryCatchPattern":null,"preventionTips":["Never reuse generic scratch buffers for scale outputs"],"tags":["quantization","fp8","scale-factor","dtype-validation"],"backgroundTag":"unsupported-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}