sgl-project/sglang · warning
FusedNormScaleShift cuda not available, using native…
Error message
FusedNormScaleShift cuda not available, using native fallback
What it means
FusedNormScaleShift's CUDA kernel requires the last dimension to be divisible by 256 and <=8192. When violated, sglang warns and uses the native (unfused) implementation — correct output, worse performance.
Solutions
- Use a hidden size that is a multiple of 256 and <=8192
- Treat as performance-only; no correctness action needed
- Profile to quantify the gap and, if needed, restructure layers to fit kernel constraints
Example fix
# before x = torch.randn(4, 1000, device="cuda") # 1000 % 256 != 0 # after x = torch.randn(4, 1024, device="cuda")
Defensive patterns
Strategy: fallback
Validate before calling
assert x.shape[-1] % 256 == 0 and x.shape[-1] <= 8192
Type guard
def fused_ok(dim: int) -> bool:
return dim % 256 == 0 and dim <= 8192 Prevention
- Validate model width against kernel constraints at config load
- Keep exotic widths out of hot paths
When it happens
Trigger: forward_cuda invoked with x.shape[-1] not a multiple of 256 or greater than 8192.
Common situations: Custom multimodal/diffusion backbones with non-standard widths; unit tests with small ad-hoc tensor sizes; importing a model config with hidden_size like 6144 is fine, but e.g. 1000 triggers it.
Related errors
- FusedScaleResidualNormScaleShift cuda not available, using…
- A CUDA VMM-enabled model must provide a multimodal processor
- All tensors must have the same data type
- attention group range
- attn_res_fused_tma requires SM100+ excluding SM12x; SM
AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28).
Data as JSON: /api/errors/1a1f139be6a5de20.
Report an issue: GitHub.
Appendix: source
Thrown at python/sglang/multimodal_gen/runtime/layers/layernorm.py:789
):
super().__init__()
self.eps = eps
if self.norm_type == "rms":
self.norm = RMSNorm(hidden_size, eps=eps, dtype=dtype)
elif self.norm_type == "layer":
self.norm = FP32LayerNorm(
hidden_size, elementwise_affine=elementwise_affine, eps=eps, dtype=dtype
)
else:
raise NotImplementedError(f"Norm type {self.norm_type} not implemented")
def forward_cuda(
self, x: torch.Tensor, shift: torch.Tensor, scale: torch.Tensor
) -> torch.Tensor:
if x.shape[-1] % 256 != 0 or x.shape[-1] > 8192:
import warnings
warnings.warn(
"FusedNormScaleShift cuda not available, using native fallback",
stacklevel=2,
)
return self.forward_native(x, shift, scale)
from sglang.kernels.ops.diffusion import fused_norm_scale_shift
return fused_norm_scale_shift(
x.contiguous(),
_ensure_contiguous(getattr(self.norm, "weight", None)),
_ensure_contiguous(getattr(self.norm, "bias", None)),
scale.contiguous(),
shift.contiguous(),
self.norm_type,
self.eps,
)
def forward_hip(View on GitHub (pinned to 0132848349)