sgl-project/sglang · error · ValueError

Output N={n_out} must be divisible by sf_vec_size={sf_vec_si

Error message

Output N={n_out} must be divisible by sf_vec_size={sf_vec_size}

What it means

The output is re-quantized to NVFP4 in groups of sf_vec_size (16) elements, so the post-SwiGLU output width n/2 must be divisible by sf_vec_size for the scale-factor tensor to have an integer number of groups. Otherwise the output scale layout would be fractional/unaligned.

Source

Thrown at python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py:2885

    major, minor = get_compute_capability(a.device)
    if major != 10:
        raise ValueError(
            f"nvfp4_gemm_swiglu_nvfp4_quant requires SM100, got SM{major}{minor}"
        )

    m = a.shape[0]
    k = a.shape[1] * 2
    n = b.shape[0]
    if b.shape[1] * 2 != k:
        raise ValueError(f"Shape mismatch: A K={k}, B K={b.shape[1] * 2}")
    if n % 2 != 0:
        raise ValueError(f"Interleaved FC1 N must be even, got {n}")

    l = 1
    n_out = n // 2
    if n_out % sf_vec_size != 0:
        raise ValueError(
            f"Output N={n_out} must be divisible by sf_vec_size={sf_vec_size}"
        )
    scale_n_out = n_out // sf_vec_size
    padded_m = _round_up(m, 128)
    padded_scale_n = _round_up(scale_n_out, 4)

    ab_dtype_cutlass = get_cutlass_dtype(ab_dtype)
    sf_dtype_cutlass = get_cutlass_dtype(sf_dtype)
    c_dtype_cutlass = get_cutlass_dtype(c_dtype)

    if m <= 128:
        mma_tiler_mn, cluster_shape_mn = (128, 128), (1, 2)
    else:
        mma_tiler_mn, cluster_shape_mn = (256, 128), (2, 1)

    if not Sm100BlockScaledPersistentDenseGemmKernel.can_implement(
        ab_dtype_cutlass,
        sf_dtype_cutlass,

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure intermediate_size is a multiple of sf_vec_size (16)
2. Keep sf_vec_size at the default 16 expected by NVFP4 kernels
3. Fall back to a non-fused path for models with unaligned intermediate size

Example fix

# before
sf_vec_size = 16  # intermediate_size = 5000 -> n_out=2500 not divisible
# after
# use intermediate_size aligned to 16, e.g. 4992 or 5016

assert (b.shape[0] // 2) % sf_vec_size == 0

def n_out_aligned(b, sf=16): return (b.shape[0]//2) % sf == 0

When it happens

Trigger: Calling nvfp4_gemm_swiglu_nvfp4_quant where (b.shape[0]//2) % sf_vec_size != 0, e.g. intermediate_size not a multiple of 16.

Common situations: Custom models with unusual intermediate sizes, or overriding sf_vec_size to a nonstandard value while the model geometry stays fixed.

Related errors

• Type must match: {self.a_dtype} != {self.b_dtype}
• dimension {dim} size {dim_size} must be divisible by 2 * gro
• nvfp4_gemm_swiglu_nvfp4_quant currently supports NVFP4 input
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• Unsupported nvfp4_gemm_swiglu_nvfp4_quant configuration: sha

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/680eb76dff0d5613. Report an issue: GitHub.