sgl-project/sglang · warning

FusedNormScaleShift cuda not available, using native fallbac

Error message

FusedNormScaleShift cuda not available, using native fallback

What it means

FusedNormScaleShift's CUDA kernel requires the last dimension to be divisible by 256 and <=8192. When violated, sglang warns and uses the native (unfused) implementation — correct output, worse performance.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/layernorm.py:789

    ):
        super().__init__()
        self.eps = eps
        if self.norm_type == "rms":
            self.norm = RMSNorm(hidden_size, eps=eps, dtype=dtype)
        elif self.norm_type == "layer":
            self.norm = FP32LayerNorm(
                hidden_size, elementwise_affine=elementwise_affine, eps=eps, dtype=dtype
            )
        else:
            raise NotImplementedError(f"Norm type {self.norm_type} not implemented")

    def forward_cuda(
        self, x: torch.Tensor, shift: torch.Tensor, scale: torch.Tensor
    ) -> torch.Tensor:
        if x.shape[-1] % 256 != 0 or x.shape[-1] > 8192:
            import warnings

            warnings.warn(
                "FusedNormScaleShift cuda not available, using native fallback",
                stacklevel=2,
            )
            return self.forward_native(x, shift, scale)

        from sglang.kernels.ops.diffusion import fused_norm_scale_shift

        return fused_norm_scale_shift(
            x.contiguous(),
            _ensure_contiguous(getattr(self.norm, "weight", None)),
            _ensure_contiguous(getattr(self.norm, "bias", None)),
            scale.contiguous(),
            shift.contiguous(),
            self.norm_type,
            self.eps,
        )

    def forward_hip(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a hidden size that is a multiple of 256 and <=8192
2. Treat as performance-only; no correctness action needed
3. Profile to quantify the gap and, if needed, restructure layers to fit kernel constraints

Example fix

# before
x = torch.randn(4, 1000, device="cuda")  # 1000 % 256 != 0
# after
x = torch.randn(4, 1024, device="cuda")

assert x.shape[-1] % 256 == 0 and x.shape[-1] <= 8192

def fused_ok(dim: int) -> bool:
    return dim % 256 == 0 and dim <= 8192

When it happens

Trigger: forward_cuda invoked with x.shape[-1] not a multiple of 256 or greater than 8192.

Common situations: Custom multimodal/diffusion backbones with non-standard widths; unit tests with small ad-hoc tensor sizes; importing a model config with hidden_size like 6144 is fine, but e.g. 1000 triggers it.

Related errors

• FusedScaleResidualNormScaleShift cuda not available, using n
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• {name} must live on CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1a1f139be6a5de20. Report an issue: GitHub.