vllm-project/vllm · error · ValueError

padded_n is not supported with TRTLLM 8x4 scale layout.

Error message

padded_n is not supported with TRTLLM 8x4 scale layout.

What it means

Raised by scaled_fp4_quant() when quantizing NVFP4 activations: the TRTLLM 8x4 scale-factor layout is auto-selected (backend contains 'trtllm' AND m <= 32 rows) while a padded_n different from n was requested. The flashinfer 8x4 kernel used for that layout cannot emit a padded output dimension, so the combination is rejected. It is an explicit TODO: the case is unimplemented, not a hardware error.

Source

Thrown at vllm/_custom_ops.py:1575

    other_dims = 1 if input.ndim == 1 else -1
    input = input.reshape(other_dims, input.shape[-1])
    m, n = input.shape
    block_size = 16

    assert n % block_size == 0, f"last dim has to be multiple of 16, but got {n}."
    assert input.dtype in (torch.float16, torch.bfloat16), (
        f"input.dtype needs to be fp16 or bf16 but got {input.dtype}."
    )
    if padded_n is not None:
        assert padded_n >= n, f"padded_n must be >= n, got padded_n={padded_n}, n={n}."
        assert padded_n % block_size == 0, (
            f"padded_n has to be a multiple of {block_size}, but got {padded_n}."
        )

    use_8x4_sf_layout = True if "trtllm" in backend and m <= 32 else False  # noqa: SIM210
    if use_8x4_sf_layout and padded_n is not None and padded_n != n:
        # TODO: support this case
        raise ValueError("padded_n is not supported with TRTLLM 8x4 scale layout.")
    if use_8x4_sf_layout:
        output, output_scale = flashinfer_quant_nvfp4_8x4_sf_layout(
            input, input_global_scale
        )
    else:
        # Pre-allocate and call .out variant (same behavior as old in-place API)
        output, output_scale = create_fp4_output_tensors(
            m,
            n,
            input.device,
            is_sf_swizzled_layout,
            padded_n=padded_n,
        )
        torch.ops._C.scaled_fp4_quant.out(
            input,
            input_global_scale,
            is_sf_swizzled_layout,
            output=output,

View on GitHub (pinned to c794754062)

Solutions

1. Pass padded_n=None (or exactly padded_n == n) when m <= 32 and backend contains 'trtllm'
2. Use a different backend string (e.g. 'none' or 'cutlass') so the 128x4 swizzled-layout path with padding is taken
3. Batch to m > 32 so the 8x4 layout is not selected and padded_n works again
4. If padding is required for the small-batch TRTLLM path, upstream support must be added (see the TODO in _custom_ops.py)

Example fix

// before
q, q_scale = ops.scaled_fp4_quant(x, gs, backend="trtllm_gen", padded_n=512)
// after (small batch decode)
q, q_scale = ops.scaled_fp4_quant(x, gs, backend="trtllm_gen", padded_n=None)

m, n = input.reshape(-1, input.shape[-1]).shape
use_8x4 = "trtllm" in backend and m <= 32
if use_8x4:
    assert padded_n is None or padded_n == n, "8x4 SF layout cannot pad; pass padded_n=None"

try:
    q, s = ops.scaled_fp4_quant(x, gs, backend=backend, padded_n=padded_n)
except ValueError as e:
    if "8x4" in str(e):
        q, s = ops.scaled_fp4_quant(x, gs, backend=backend, padded_n=None)
    else:
        raise

When it happens

Trigger: Calling vllm._custom_ops.scaled_fp4_quant(input, global_scale, backend='trtllm' or e.g. 'trtllm_gen', padded_n=N) with input reshaped to m<=32 rows (single token / small batch) and padded_n != n.

Common situations: Running a TRTLLM-gen quantized model (nvfp4 quant config) during decode with batch size <= 32 where a downstream GEMM wants K padded to a larger alignment; a padded_n larger than n is passed for cublas/trtllm gemm compatibility.

Related errors

• `a` must have at least 1 dimension.
• last dim of `a` must be divisible by 32, got {a.size(-1)}.
• `a` and `b` must be on the same device.
• invalid method {method!r}, must be 'quest' or 'abs_max'
• 'mm_encoder_fp8_scale_path' and 'mm_encoder_fp8_scale_save_p

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f5fde618b2055bfa. Report an issue: GitHub.