{"record":{"id":"5e8d813665340329","repo":"vllm-project/vllm","slug":"last-dim-of-a-must-be-divisible-by-32-got-a-si","errorCode":null,"errorMessage":"last dim of `a` must be divisible by 32, got {a.size(-1)}.","messagePattern":"last dim of `a` must be divisible by 32, got (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/_custom_ops.py","lineNumber":4098,"sourceCode":"        return xh_e2m1, xh_e8m0\n\n\nif hasattr(torch.ops._qutlass_C, \"fusedQuantizeMxAbsMax\"):\n\n    @register_fake(\"_qutlass_C::fusedQuantizeMxAbsMax\")\n    def _fake_fused_quantize_mx_absmax(\n        a: torch.Tensor, b: torch.Tensor, xh_e2m1: torch.Tensor, xh_e8m0: torch.Tensor\n    ):\n        return xh_e2m1, xh_e8m0\n\n\ndef fusedQuantizeMx(\n    a: torch.Tensor, b: torch.Tensor, *, method: Literal[\"quest\", \"abs_max\"] = \"quest\"\n) -> tuple[torch.Tensor, torch.Tensor]:\n    if a.dim() == 0:\n        raise ValueError(\"`a` must have at least 1 dimension.\")\n    if a.size(-1) % 32 != 0:\n        raise ValueError(f\"last dim of `a` must be divisible by 32, got {a.size(-1)}.\")\n    if b.device != a.device:\n        raise ValueError(\"`a` and `b` must be on the same device.\")\n\n    xh_e2m1 = torch.empty(\n        *a.shape[:-1], a.size(-1) // 2, dtype=torch.uint8, device=a.device\n    )\n\n    rows, cols = a.numel() // a.size(-1), a.size(-1) // 32\n    n_row_blocks = cdiv(rows, 128)\n    n_col_blocks = cdiv(cols, 4)\n    padded_rows = n_row_blocks * 128\n    padded_cols = n_col_blocks * 4\n\n    xh_e8m0 = torch.empty(\n        padded_rows, padded_cols, dtype=torch.float8_e8m0fnu, device=a.device\n    )\n\n    if not hasattr(torch.ops, \"_qutlass_C\"):","sourceCodeStart":4080,"sourceCodeEnd":4116,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/_custom_ops.py#L4080-L4116","documentation":"fusedQuantizeMx() quantizes the last dimension of `a` into MX blocks of 32 elements (two FP4 values per byte, one e8m0 scale per 32 elements). If a.size(-1) is not divisible by 32 the block decomposition is impossible, so it raises ValueError before allocating outputs.","triggerScenarios":"Calling vllm._custom_ops.fusedQuantizeMx(a, b) where the hidden/last dim of a is not a multiple of 32 (e.g. hidden_size 6144 works, 6150 fails; a sliced tensor like x[:, :1000] fails).","commonSituations":"Custom models with unusual hidden sizes; slicing projections/tails off KV caches or keys before MX quantization in quest sparse-attention; off-by-one off-by-few slicing bugs that break alignment.","solutions":["Pad the last dim to the next multiple of 32 (torch.nn.functional.pad) and slice back after quantization if needed","Fix the slicing/indexing bug that produced a non-multiple-of-32 last dim","Choose a model/config whose hidden dim is a multiple of 32 (virtually all standard transformers are)"],"exampleFix":"# before\nq, s = ops.fusedQuantizeMx(keys[:, :1000], b)\n# after\npad = (-1000) % 32\nq, s = ops.fusedQuantizeMx(torch.nn.functional.pad(keys[:, :1000], (0, pad)), b)","handlingStrategy":"validation","validationCode":"last = a.size(-1)\nassert last % 32 == 0, f\"last dim must be % 32, got {last}\"","typeGuard":"def mx_aligned(a: torch.Tensor) -> bool:\n    return a.dim() >= 1 and a.size(-1) % 32 == 0","tryCatchPattern":null,"preventionTips":["Pad to 32 before quantization, slice after","Add shape asserts at module boundaries instead of trusting upstream slicing"],"tags":["quantization","mx-format","fp4","validation","shape","alignment"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}