{"record":{"id":"393d1e86409ab4b0","repo":"sgl-project/sglang","slug":"unsupported-residual-gate-add-dtype-dtype","errorCode":null,"errorMessage":"Unsupported residual_gate_add dtype: {dtype}","messagePattern":"Unsupported residual_gate_add dtype: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/diffusion/modulate/residual_gate_add_jit.py","lineNumber":25,"sourceCode":"\nfrom sglang.kernels.jit.utils import cache_once, load_jit, make_cpp_args\nfrom sglang.srt.utils.custom_op import register_custom_op\n\nif TYPE_CHECKING:\n    from tvm_ffi.module import Module\n\n\n_SUPPORTED_DTYPES = (torch.float16, torch.bfloat16, torch.float32)\n_BIT_EXACT_DTYPES = (torch.float16, torch.bfloat16)\n_FAILED_RUNTIME_KEYS: set[tuple[int | None, torch.dtype]] = set()\n\nlogger = logging.getLogger(__name__)\n\n\n@cache_once\ndef _jit_residual_gate_add_module(dtype: torch.dtype) -> Module:\n    if dtype not in _SUPPORTED_DTYPES:\n        raise RuntimeError(f\"Unsupported residual_gate_add dtype: {dtype}\")\n    args = make_cpp_args(dtype)\n    return load_jit(\n        \"diffusion_residual_gate_add\",\n        *args,\n        cuda_files=[\"diffusion/residual_gate_add.cuh\"],\n        cuda_wrappers=[\n            (\n                \"residual_gate_add\",\n                \"residual_gate_add::\" f\"ResidualGateAddKernel<{args}>::run\",\n            ),\n        ],\n    )\n\n\ndef _fake_impl(\n    residual: torch.Tensor, update: torch.Tensor, gate: torch.Tensor\n) -> torch.Tensor:\n    return torch.empty_like(residual)","sourceCodeStart":7,"sourceCodeEnd":43,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/diffusion/modulate/residual_gate_add_jit.py#L7-L43","documentation":"The JIT-compiled residual_gate_add kernel supports only the dtypes in its _SUPPORTED_DTYPES (fp16/bf16/fp32). _jit_residual_gate_add_module raises when the requested dtype is outside that set, preventing compilation of an unsupported variant.","triggerScenarios":"Calling _residual_gate_add_custom_op (via the residual gate add CUDA path) with tensors in fp64, fp8, or integer dtype; the dtype is used as the JIT specialization key and rejected.","commonSituations":"Diffusion transformer residual paths where activations were promoted to float64 (e.g. by a norm in fp32 not cast back), or partially-quantized fp8 pipelines routing through the fused residual add.","solutions":["Cast the residual and gated tensors to bf16/fp16/fp32 before the fused call","Use the eager expression (residual + gate * x style) as fallback","Audit the preceding norm/modulate ops for unwanted dtype promotion","Extend the JIT kernel and _SUPPORTED_DTYPES if a new dtype is truly needed"],"exampleFix":"# before\ny = _residual_gate_add_custom_op(x_fp64, res_fp64)\n# after\nx = x.to(torch.bfloat16); res = res.to(torch.bfloat16)\ny = _residual_gate_add_custom_op(x, res)","handlingStrategy":"type-guard","validationCode":"if x.dtype not in (torch.float16, torch.bfloat16, torch.float32):\n    x = x.to(torch.bfloat16); residual = residual.to(torch.bfloat16)","typeGuard":"def gate_dtype_ok(x: torch.Tensor) -> bool:\n    return x.dtype in (torch.float16, torch.bfloat16, torch.float32)","tryCatchPattern":null,"preventionTips":["Check for fp64 promotion after norms","Fall back to eager residual add for unsupported dtypes"],"tags":["dtype","jit","residual","cuda"],"backgroundTag":"unsupported-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}