sgl-project/sglang · error · ValueError

The last dimension ({input.shape[-1]}) x itemsize ({input.dt

Error message

The last dimension ({input.shape[-1]}) x itemsize ({input.dtype.itemsize}) must be a multiple of 16 bytes.

What it means

gelu_quick (QuickGELU, y = x*sigmoid(1.702x)) requires the last dimension in bytes to be a multiple of 16 because the CUDA/HIP kernel does 128-bit vectorized loads/stores. Unlike the *_and_mul variants it reports the dimension, itemsize, and computed requirement explicitly.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/elementwise.py:313

            input.shape[:-1] + (input.shape[-1] // 2,),
            device=input.device,
            dtype=input.dtype,
        )
    torch.ops.sgl_kernel.gelu_and_mul.default(out, input)
    return out


if torch.version.hip is not None:

    def gelu_quick(input: torch.Tensor, out: torch.Tensor = None) -> torch.Tensor:
        """
        Quick-GELU:  y = x * sigmoid(1.702 * x)

        The CUDA/HIP kernel uses 128-bit (16-byte) vector loads & stores,
        so the last-dimension byte length must be a multiple of 16 bytes.
        """
        if input.shape[-1] * input.dtype.itemsize % 16 != 0:
            raise ValueError(
                f"The last dimension ({input.shape[-1]}) x itemsize "
                f"({input.dtype.itemsize}) must be a multiple of 16 bytes."
            )

        if out is not None:
            assert input.shape == out.shape, f"{input.shape} != {out.shape}"
        else:
            out = torch.empty_like(input)

        torch.ops.sgl_kernel.gelu_quick(out, input)
        return out


def dsv4_fused_q_norm_rope(
    q_input: torch.Tensor,
    freqs_cis: torch.Tensor,
    positions: torch.Tensor,
    eps: float = 1e-6,

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a feature width aligned to 16 bytes (bf16/fp16: %8==0, fp32: %4==0).
2. Pad last dim to alignment and slice after: gelu_quick(pad(x))[..., :d].
3. Fall back to x * torch.sigmoid(1.702*x).

Example fix

# before
y = gelu_quick(x)  # x fp32, last dim 7
# after
y = gelu_quick(torch.nn.functional.pad(x, (0,1)))[..., :7]

if input.shape[-1] * input.dtype.itemsize % 16 != 0:
    input = pad_to_16b(input)

def gelu_quick_ready(x: torch.Tensor) -> bool:
    return x.shape[-1] * x.dtype.itemsize % 16 == 0

try:
    y = gelu_quick(x)
except ValueError:
    y = x * torch.sigmoid(1.702 * x)

When it happens

Trigger: Calling gelu_quick with last-dim * itemsize % 16 != 0 — e.g. fp32 tensor of width 7, bf16 width 12; Qwen-style QuickGELU MLPs with non-aligned sizes (Qwen sizes are normally aligned).

Common situations: Testing the kernel with tiny toy tensors; custom architectures with odd feature widths.

Related errors

• The pointers must be multiple of 16 bytes.
• rmsnorm_hf: unsupported hidden_size={hidden_size} (must be a
• camera_conditions must have shape (T,20) or (B,T,20), got {t
• chunk_plucker must have shape (48,T,H,W) or (B,48,T,H,W), go
• f"recurrent_kda state pool breaks the compiled stride contra

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f2ae37c4ef5322ec. Report an issue: GitHub.