vllm-project/vllm · error · ValueError

last dim of `a` must be divisible by 32, got {a.size(-1)}.

Error message

last dim of `a` must be divisible by 32, got {a.size(-1)}.

What it means

fusedQuantizeMx() quantizes the last dimension of `a` into MX blocks of 32 elements (two FP4 values per byte, one e8m0 scale per 32 elements). If a.size(-1) is not divisible by 32 the block decomposition is impossible, so it raises ValueError before allocating outputs.

Source

Thrown at vllm/_custom_ops.py:4098

        return xh_e2m1, xh_e8m0


if hasattr(torch.ops._qutlass_C, "fusedQuantizeMxAbsMax"):

    @register_fake("_qutlass_C::fusedQuantizeMxAbsMax")
    def _fake_fused_quantize_mx_absmax(
        a: torch.Tensor, b: torch.Tensor, xh_e2m1: torch.Tensor, xh_e8m0: torch.Tensor
    ):
        return xh_e2m1, xh_e8m0


def fusedQuantizeMx(
    a: torch.Tensor, b: torch.Tensor, *, method: Literal["quest", "abs_max"] = "quest"
) -> tuple[torch.Tensor, torch.Tensor]:
    if a.dim() == 0:
        raise ValueError("`a` must have at least 1 dimension.")
    if a.size(-1) % 32 != 0:
        raise ValueError(f"last dim of `a` must be divisible by 32, got {a.size(-1)}.")
    if b.device != a.device:
        raise ValueError("`a` and `b` must be on the same device.")

    xh_e2m1 = torch.empty(
        *a.shape[:-1], a.size(-1) // 2, dtype=torch.uint8, device=a.device
    )

    rows, cols = a.numel() // a.size(-1), a.size(-1) // 32
    n_row_blocks = cdiv(rows, 128)
    n_col_blocks = cdiv(cols, 4)
    padded_rows = n_row_blocks * 128
    padded_cols = n_col_blocks * 4

    xh_e8m0 = torch.empty(
        padded_rows, padded_cols, dtype=torch.float8_e8m0fnu, device=a.device
    )

    if not hasattr(torch.ops, "_qutlass_C"):

View on GitHub (pinned to c794754062)

Solutions

1. Pad the last dim to the next multiple of 32 (torch.nn.functional.pad) and slice back after quantization if needed
2. Fix the slicing/indexing bug that produced a non-multiple-of-32 last dim
3. Choose a model/config whose hidden dim is a multiple of 32 (virtually all standard transformers are)

Example fix

# before
q, s = ops.fusedQuantizeMx(keys[:, :1000], b)
# after
pad = (-1000) % 32
q, s = ops.fusedQuantizeMx(torch.nn.functional.pad(keys[:, :1000], (0, pad)), b)

last = a.size(-1)
assert last % 32 == 0, f"last dim must be % 32, got {last}"

def mx_aligned(a: torch.Tensor) -> bool:
    return a.dim() >= 1 and a.size(-1) % 32 == 0

When it happens

Trigger: Calling vllm._custom_ops.fusedQuantizeMx(a, b) where the hidden/last dim of a is not a multiple of 32 (e.g. hidden_size 6144 works, 6150 fails; a sliced tensor like x[:, :1000] fails).

Common situations: Custom models with unusual hidden sizes; slicing projections/tails off KV caches or keys before MX quantization in quest sparse-attention; off-by-one off-by-few slicing bugs that break alignment.

Related errors

• `a` must have at least 1 dimension.
• `a` and `b` must be on the same device.
• invalid method {method!r}, must be 'quest' or 'abs_max'
• padded_n is not supported with TRTLLM 8x4 scale layout.
• 'mm_encoder_fp8_scale_path' and 'mm_encoder_fp8_scale_save_p

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/5e8d813665340329. Report an issue: GitHub.