{"record":{"id":"7ac354579fc76f99","repo":"sgl-project/sglang","slug":"fusedscaleresidualnormscaleshift-cuda-not-availabl","errorCode":null,"errorMessage":"FusedScaleResidualNormScaleShift cuda not available, using native fallback","messagePattern":"FusedScaleResidualNormScaleShift cuda not available, using native fallback","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/multimodal_gen/runtime/layers/layernorm.py","lineNumber":613,"sourceCode":"            )\n        else:\n            raise NotImplementedError(f\"Norm type {self.norm_type} not implemented\")\n\n    def forward_cuda(\n        self,\n        residual: torch.Tensor,\n        x: torch.Tensor,\n        gate: torch.Tensor | int,\n        shift: torch.Tensor,\n        scale: torch.Tensor,\n    ) -> tuple[torch.Tensor, torch.Tensor]:\n        if residual.numel() == 0 or x.numel() == 0:\n            return self.forward_native(residual, x, gate, shift, scale)\n\n        if x.shape[-1] % 256 != 0 or x.shape[-1] > 8192:\n            import warnings\n\n            warnings.warn(\n                \"FusedScaleResidualNormScaleShift cuda not available, using native fallback\",\n                stacklevel=2,\n            )\n            return self.forward_native(residual, x, gate, shift, scale)\n\n        from sglang.kernels.ops.diffusion import fused_scale_residual_norm_scale_shift\n\n        if isinstance(gate, int) and gate != 1:\n            raise ValueError(\n                f\"Only gate value of 1 is supported for int type, but got {gate}\"\n            )\n\n        return fused_scale_residual_norm_scale_shift(\n            residual.contiguous(),\n            x.contiguous(),\n            gate.contiguous() if isinstance(gate, torch.Tensor) else None,\n            _ensure_contiguous(getattr(self.norm, \"weight\", None)),\n            _ensure_contiguous(getattr(self.norm, \"bias\", None)),","sourceCodeStart":595,"sourceCodeEnd":631,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/layernorm.py#L595-L631","documentation":"The fused CUDA kernel for ScaleResidualNorm+ScaleShift only supports hidden dims divisible by 256 and at most 8192. Outside that range sglang falls back to a slower native PyTorch implementation and warns; results are correct, only performance changes.","triggerScenarios":"Calling forward_cuda on a model whose last tensor dimension x.shape[-1] is not a multiple of 256 or exceeds 8192, or with empty tensors.","commonSituations":"Diffusion/multimodal models with unusual hidden sizes; custom width configurations; testing tiny shapes locally; FP8 branch also falls back (CUDA capability < 9.0).","solutions":["Align the hidden dimension to a multiple of 256 within the <=8192 limit if you control the architecture","Accept the fallback for exotic shapes — it is functionally correct","Benchmark both paths; if the fallback dominates latency, reshape/pad when mathematically safe"],"exampleFix":"# before\nlayer = FusedScaleResidualNormScaleShift(7680)  # 7680 % 256 != 0\n# after\nlayer = FusedScaleResidualNormScaleShift(7680 // 256 * 256)  # or accept fallback","handlingStrategy":"fallback","validationCode":"assert x.shape[-1] % 256 == 0 and x.shape[-1] <= 8192, \"fused kernel unavailable; native fallback will run\"","typeGuard":"def fused_ok(dim: int) -> bool:\n    return dim % 256 == 0 and dim <= 8192","tryCatchPattern":null,"preventionTips":["Choose hidden sizes divisible by 256","Benchmark fallback shapes before committing an architecture"],"tags":["cuda","kernel-fallback","layernorm","performance","shape-constraint"],"backgroundTag":"kernel-shape-fallback","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}