sgl-project/sglang · warning

FusedScaleResidualNormScaleShift cuda not available, using n

Error message

FusedScaleResidualNormScaleShift cuda not available, using native fallback

What it means

The fused CUDA kernel for ScaleResidualNorm+ScaleShift only supports hidden dims divisible by 256 and at most 8192. Outside that range sglang falls back to a slower native PyTorch implementation and warns; results are correct, only performance changes.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/layernorm.py:613

            )
        else:
            raise NotImplementedError(f"Norm type {self.norm_type} not implemented")

    def forward_cuda(
        self,
        residual: torch.Tensor,
        x: torch.Tensor,
        gate: torch.Tensor | int,
        shift: torch.Tensor,
        scale: torch.Tensor,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        if residual.numel() == 0 or x.numel() == 0:
            return self.forward_native(residual, x, gate, shift, scale)

        if x.shape[-1] % 256 != 0 or x.shape[-1] > 8192:
            import warnings

            warnings.warn(
                "FusedScaleResidualNormScaleShift cuda not available, using native fallback",
                stacklevel=2,
            )
            return self.forward_native(residual, x, gate, shift, scale)

        from sglang.kernels.ops.diffusion import fused_scale_residual_norm_scale_shift

        if isinstance(gate, int) and gate != 1:
            raise ValueError(
                f"Only gate value of 1 is supported for int type, but got {gate}"
            )

        return fused_scale_residual_norm_scale_shift(
            residual.contiguous(),
            x.contiguous(),
            gate.contiguous() if isinstance(gate, torch.Tensor) else None,
            _ensure_contiguous(getattr(self.norm, "weight", None)),
            _ensure_contiguous(getattr(self.norm, "bias", None)),

View on GitHub (pinned to 0132848349)

Solutions

1. Align the hidden dimension to a multiple of 256 within the <=8192 limit if you control the architecture
2. Accept the fallback for exotic shapes — it is functionally correct
3. Benchmark both paths; if the fallback dominates latency, reshape/pad when mathematically safe

Example fix

# before
layer = FusedScaleResidualNormScaleShift(7680)  # 7680 % 256 != 0
# after
layer = FusedScaleResidualNormScaleShift(7680 // 256 * 256)  # or accept fallback

assert x.shape[-1] % 256 == 0 and x.shape[-1] <= 8192, "fused kernel unavailable; native fallback will run"

def fused_ok(dim: int) -> bool:
    return dim % 256 == 0 and dim <= 8192

When it happens

Trigger: Calling forward_cuda on a model whose last tensor dimension x.shape[-1] is not a multiple of 256 or exceeds 8192, or with empty tensors.

Common situations: Diffusion/multimodal models with unusual hidden sizes; custom width configurations; testing tiny shapes locally; FP8 branch also falls back (CUDA capability < 9.0).

Related errors

• FusedNormScaleShift cuda not available, using native fallbac
• Cannot find NVIDIA Math-DX (cuBLASDx) headers. Install the `
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• {name} must live on CUDA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7ac354579fc76f99. Report an issue: GitHub.