{"record":{"id":"1a1f139be6a5de20","repo":"sgl-project/sglang","slug":"fusednormscaleshift-cuda-not-available-using-nati","errorCode":null,"errorMessage":"FusedNormScaleShift cuda not available, using native fallback","messagePattern":"FusedNormScaleShift cuda not available, using native fallback","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/multimodal_gen/runtime/layers/layernorm.py","lineNumber":789,"sourceCode":"    ):\n        super().__init__()\n        self.eps = eps\n        if self.norm_type == \"rms\":\n            self.norm = RMSNorm(hidden_size, eps=eps, dtype=dtype)\n        elif self.norm_type == \"layer\":\n            self.norm = FP32LayerNorm(\n                hidden_size, elementwise_affine=elementwise_affine, eps=eps, dtype=dtype\n            )\n        else:\n            raise NotImplementedError(f\"Norm type {self.norm_type} not implemented\")\n\n    def forward_cuda(\n        self, x: torch.Tensor, shift: torch.Tensor, scale: torch.Tensor\n    ) -> torch.Tensor:\n        if x.shape[-1] % 256 != 0 or x.shape[-1] > 8192:\n            import warnings\n\n            warnings.warn(\n                \"FusedNormScaleShift cuda not available, using native fallback\",\n                stacklevel=2,\n            )\n            return self.forward_native(x, shift, scale)\n\n        from sglang.kernels.ops.diffusion import fused_norm_scale_shift\n\n        return fused_norm_scale_shift(\n            x.contiguous(),\n            _ensure_contiguous(getattr(self.norm, \"weight\", None)),\n            _ensure_contiguous(getattr(self.norm, \"bias\", None)),\n            scale.contiguous(),\n            shift.contiguous(),\n            self.norm_type,\n            self.eps,\n        )\n\n    def forward_hip(","sourceCodeStart":771,"sourceCodeEnd":807,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/layernorm.py#L771-L807","documentation":"FusedNormScaleShift's CUDA kernel requires the last dimension to be divisible by 256 and <=8192. When violated, sglang warns and uses the native (unfused) implementation — correct output, worse performance.","triggerScenarios":"forward_cuda invoked with x.shape[-1] not a multiple of 256 or greater than 8192.","commonSituations":"Custom multimodal/diffusion backbones with non-standard widths; unit tests with small ad-hoc tensor sizes; importing a model config with hidden_size like 6144 is fine, but e.g. 1000 triggers it.","solutions":["Use a hidden size that is a multiple of 256 and <=8192","Treat as performance-only; no correctness action needed","Profile to quantify the gap and, if needed, restructure layers to fit kernel constraints"],"exampleFix":"# before\nx = torch.randn(4, 1000, device=\"cuda\")  # 1000 % 256 != 0\n# after\nx = torch.randn(4, 1024, device=\"cuda\")","handlingStrategy":"fallback","validationCode":"assert x.shape[-1] % 256 == 0 and x.shape[-1] <= 8192","typeGuard":"def fused_ok(dim: int) -> bool:\n    return dim % 256 == 0 and dim <= 8192","tryCatchPattern":null,"preventionTips":["Validate model width against kernel constraints at config load","Keep exotic widths out of hot paths"],"tags":["cuda","kernel-fallback","layernorm","performance","shape-constraint"],"backgroundTag":"kernel-shape-fallback","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}