sgl-project/sglang · error · ValueError

The pointers must be multiple of 16 bytes.

Error message

The pointers must be multiple of 16 bytes.

What it means

silu_and_mul validates that input.shape[-1] * input.dtype.itemsize is a multiple of 16 bytes because the CUDA kernel uses 128-bit vectorized loads. The message says 'pointers' but the actual check is on the last-dimension byte size; e.g. bf16 input with an odd hidden size (2*h bytes not divisible by 16) fails.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/elementwise.py:260

            input, residual, weight, eps, enable_pdl
        )
    else:
        _gemma_fused_add_rmsnorm_internal(input, residual, weight, eps, enable_pdl)


def _check_shape(input: torch.Tensor, output: torch.Tensor) -> None:
    assert input.ndim == output.ndim, f"{input.ndim} != {output.ndim}"
    assert (
        input.shape[:-1] == output.shape[:-1]
    ), f"{input.shape[:-1]} != {output.shape[:-1]}"
    assert (
        input.shape[-1] == 2 * output.shape[-1]
    ), f"{input.shape[-1]} != {2 * output.shape[-1]}"


def silu_and_mul(input: torch.Tensor, out: torch.Tensor = None) -> torch.Tensor:
    if input.shape[-1] * input.dtype.itemsize % 16 != 0:
        raise ValueError("The pointers must be multiple of 16 bytes.")
    if out is not None:
        _check_shape(input, out)
    else:
        out = torch.empty(
            input.shape[:-1] + (input.shape[-1] // 2,),
            device=input.device,
            dtype=input.dtype,
        )
    torch.ops.sgl_kernel.silu_and_mul.default(out, input)
    return out


def gelu_tanh_and_mul(input: torch.Tensor, out: torch.Tensor = None) -> torch.Tensor:
    if input.shape[-1] * input.dtype.itemsize % 16 != 0:
        raise ValueError("The pointers must be multiple of 16 bytes.")
    if out is not None:
        _check_shape(input, out)
    else:

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a hidden size whose byte length is 16-byte aligned (bf16: multiple of 8 elements; fp16: multiple of 8; fp32: multiple of 4).
2. Pad the last dimension to alignment before the call and slice afterwards if exactness matters.
3. Use the PyTorch-native fallback (torch.nn.functional.silu) for non-aligned shapes.

Example fix

# before
out = silu_and_mul(x)  # x: bf16, shape[-1] = 5000 (10000 bytes, not %16)
# after
x = torch.nn.functional.pad(x, (0, 8))
out = silu_and_mul(x)[..., :2500]

def silu_mul_ready(x: torch.Tensor) -> bool:
    return x.shape[-1] * x.dtype.itemsize % 16 == 0
assert silu_mul_ready(x)

def silu_mul_ready(x: torch.Tensor) -> bool:
    return x.dim() > 0 and x.shape[-1] * x.dtype.itemsize % 16 == 0

try:
    out = silu_and_mul(x)
except ValueError:
    out = torch.nn.functional.silu(x[..., :d]) * x[..., d:]  # fallback

When it happens

Trigger: Calling sgl_kernel.elementwise.silu_and_mul with hidden dim * itemsize % 16 != 0, e.g. fp32 with d%4!=0 or bf16 with d%8!=0; using a custom model whose intermediate size after the concat is not 16-byte aligned in bytes.

Common situations: Porting a model with an unusual MLP intermediate size; testing kernels with toy tensors of tiny/odd sizes.

Related errors

• The last dimension ({input.shape[-1]}) x itemsize ({input.dt
• rmsnorm_hf: unsupported hidden_size={hidden_size} (must be a
• camera_conditions must have shape (T,20) or (B,T,20), got {t
• chunk_plucker must have shape (48,T,H,W) or (B,48,T,H,W), go
• f"recurrent_kda state pool breaks the compiled stride contra

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cb91e183b637c9cb. Report an issue: GitHub.