What causes "last dim of `a` must be divisible by 32, got {a.size(-1)}." in vllm-project/vllm?

Pad the last dim to the next multiple of 32 (torch.nn.functional.pad) and slice back after quantization if needed

vllm-project/vllm · error · ValueError

last dim of `a` must be divisible by 32, got

Error message

last dim of `a` must be divisible by 32, got {a.size(-1)}.

What it means

fusedQuantizeMx() quantizes the last dimension of `a` into MX blocks of 32 elements (two FP4 values per byte, one e8m0 scale per 32 elements). If a.size(-1) is not divisible by 32 the block decomposition is impossible, so it raises ValueError before allocating outputs.

Solutions

Pad the last dim to the next multiple of 32 (torch.nn.functional.pad) and slice back after quantization if needed
Fix the slicing/indexing bug that produced a non-multiple-of-32 last dim
Choose a model/config whose hidden dim is a multiple of 32 (virtually all standard transformers are)

Example fix

# before
q, s = ops.fusedQuantizeMx(keys[:, :1000], b)
# after
pad = (-1000) % 32
q, s = ops.fusedQuantizeMx(torch.nn.functional.pad(keys[:, :1000], (0, pad)), b)

Defensive patterns

Strategy: validation

Validate before calling

last = a.size(-1)
assert last % 32 == 0, f"last dim must be % 32, got {last}"

Type guard

def mx_aligned(a: torch.Tensor) -> bool:
    return a.dim() >= 1 and a.size(-1) % 32 == 0

Prevention

Pad to 32 before quantization, slice after
Add shape asserts at module boundaries instead of trusting upstream slicing

When it happens

Trigger: Calling vllm._custom_ops.fusedQuantizeMx(a, b) where the hidden/last dim of a is not a multiple of 32 (e.g. hidden_size 6144 works, 6150 fails; a sliced tensor like x[:, :1000] fails).

Common situations: Custom models with unusual hidden sizes; slicing projections/tails off KV caches or keys before MX quantization in quest sparse-attention; off-by-one off-by-few slicing bugs that break alignment.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/5e8d813665340329. Report an issue: GitHub.

Appendix: source

Thrown at vllm/_custom_ops.py:4098

        return xh_e2m1, xh_e8m0


if hasattr(torch.ops._qutlass_C, "fusedQuantizeMxAbsMax"):

    @register_fake("_qutlass_C::fusedQuantizeMxAbsMax")
    def _fake_fused_quantize_mx_absmax(
        a: torch.Tensor, b: torch.Tensor, xh_e2m1: torch.Tensor, xh_e8m0: torch.Tensor
    ):
        return xh_e2m1, xh_e8m0


def fusedQuantizeMx(
    a: torch.Tensor, b: torch.Tensor, *, method: Literal["quest", "abs_max"] = "quest"
) -> tuple[torch.Tensor, torch.Tensor]:
    if a.dim() == 0:
        raise ValueError("`a` must have at least 1 dimension.")
    if a.size(-1) % 32 != 0:
        raise ValueError(f"last dim of `a` must be divisible by 32, got {a.size(-1)}.")
    if b.device != a.device:
        raise ValueError("`a` and `b` must be on the same device.")

    xh_e2m1 = torch.empty(
        *a.shape[:-1], a.size(-1) // 2, dtype=torch.uint8, device=a.device
    )

    rows, cols = a.numel() // a.size(-1), a.size(-1) // 32
    n_row_blocks = cdiv(rows, 128)
    n_col_blocks = cdiv(cols, 4)
    padded_rows = n_row_blocks * 128
    padded_cols = n_col_blocks * 4

    xh_e8m0 = torch.empty(
        padded_rows, padded_cols, dtype=torch.float8_e8m0fnu, device=a.device
    )

    if not hasattr(torch.ops, "_qutlass_C"):

View on GitHub (pinned to c794754062)