sgl-project/sglang · error · ValueError

D={D} not supported, must be multiple of 256 and <= 8192

Error message

D={D} not supported, must be multiple of 256 and <= 8192

What it means

The CuTe DSL kernel tiles the hidden dimension in 256-wide blocks and is only compiled for D up to 8192. fused_norm_scale_shift rejects hidden sizes that are not a multiple of 256 or exceed 8192.

Source

Thrown at python/sglang/kernels/ops/diffusion/norm/scale_residual_norm_cutedsl.py:287

    native_y = _try_qwen_native_norm_scale_shift(
        x, weight, bias, scale, shift, norm_type, eps
    )
    if native_y is not None:
        return native_y
    stream = cuda.CUstream(torch.cuda.current_stream().cuda_stream)
    # Tensor Validation
    BSD = x.shape
    validate_x(x, *BSD)
    validate_weight_bias(weight, BSD[-1])
    validate_weight_bias(bias, BSD[-1])
    validate_scale_shift(scale, *BSD)
    validate_scale_shift(shift, *BSD)

    if norm_type == "layer" or norm_type == "rms":
        D = x.shape[-1]
        if D % 256 != 0 or D > 8192:
            raise ValueError(
                f"D={D} not supported, must be multiple of 256 and <= 8192"
            )
        y = torch.empty_like(x)  # create output tensor
        scale = broadcast_tensor_for_bsfd(scale, *x.shape)  # handle various shapes
        shift = broadcast_tensor_for_bsfd(shift, *x.shape)  # handle various shapes
        # Use scalar placeholders for None tensors as a workaround, since the CuTe DSL
        # TVM-FFI backend does not support None parameters. scalar values do not result
        # in code generation and have no impact on runtime performance.
        weight = 1 if weight is None else weight
        bias = 0 if bias is None else bias
        ResOut, Residual, Gate = 0, 0, 1
        torch_tensors = [y, ResOut, Residual, x, Gate, weight, bias, scale, shift]
        # Compile cache
        hash_key = ScaleResidualNormScaleShift.make_hash_key(norm_type, *torch_tensors)
        compiled_fn = _COMPILE_CACHE.get(hash_key)
        if compiled_fn is None:
            kernel = ScaleResidualNormScaleShift(D, norm_type)
            fake_sig_args = [to_fake_cute_args(t) for t in torch_tensors]

View on GitHub (pinned to 0132848349)

Solutions

1. Route such shapes to the eager torch.nn.functional.layer_norm / RMSNorm fallback
2. Pad D to the next multiple of 256 if padding is semantically acceptable
3. Reconfigure the model's hidden size to a multiple of 256 (design-time fix)

Example fix

# before
y = fused_norm_scale_shift(x, w, b, scale, shift, "rms")  # D=1000
# after
if D % 256 == 0 and D <= 8192:
    y = fused_norm_scale_shift(x, w, b, scale, shift, "rms")
else:
    y = torch.nn.functional.rms_norm(x, (D,), w, eps)  # fallback

D = x.shape[-1]
use_fused = (D % 256 == 0 and D <= 8192)

def d_supported(D: int) -> bool:
    return D % 256 == 0 and D <= 8192

try:
    y = fused_norm_scale_shift(...)
except ValueError:
    y = torch.nn.functional.layer_norm(x * scale + shift if False else x, (D,), weight, bias, eps)  # eager fallback

When it happens

Trigger: Calling fused_norm_scale_shift with norm_type 'layer' or 'rms' when x.shape[-1] (D) is not divisible by 256 or is > 8192 (e.g. D=1000, D=6144, or D=16384).

Common situations: Small test models with odd hidden sizes, or very wide MLP hidden dims routed through this fused path by mistake.

Related errors

• Validate failed: unsupported dtype: {t.dtype}
• Validate failed: unsupported tensor shape: {t.shape}.
• unknown norm_type {norm_type}
• No frames were recorded
• Unsupported content type ${header.content_type}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/76fc8719ee2c669a. Report an issue: GitHub.