huggingface/transformers · error · ValueError

Weight shape ({rows}, {cols}) not divisible by scale grid ({

Error message

Weight shape ({rows}, {cols}) not divisible by scale grid ({scale_rows}, {scale_cols}).

What it means

In the FP8/MXFP4 dequantize path, the weight's last two dimensions must tile exactly by the scale grid: rows % scale_rows and cols % scale_cols must both be 0. The block size is derived from scales.shape[-2:] (MoE MXFP4 experts use a [1,32] grid, dense FP8 linears use [128,128]); a weight whose shape is not an integer multiple of that grid means the scale tensor does not correspond to this weight, so dequantization is refused rather than producing garbage. Note the fallback that treats an unwrapped single scale as a [1,1] grid.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:1025

    ) -> torch.Tensor:
        # FP4 path: int8 / float4_e2m1fn_x2 stores two nibbles per byte. Unpack to fp32
        # first so the rest of the routine sees a normal (rows, cols) float matrix.
        fp4_dtype = getattr(torch, "float4_e2m1fn_x2", None)
        if quantized.dtype == torch.int8 or (fp4_dtype is not None and quantized.dtype == fp4_dtype):
            quantized_fp32 = self._unpack_fp4(quantized)
        else:
            quantized_fp32 = quantized.to(torch.float32)
        rows, cols = quantized_fp32.shape[-2:]
        # Derive block size from the scale grid rather than the global config: MoE experts
        # ship MXFP4 with a ``[1, 32]`` block, dense linears ship FP8 with ``[128, 128]``,
        # and the same dequant has to handle both within one checkpoint.
        try:
            scale_rows, scale_cols = scales.shape[-2:]
        except Exception:
            # scale can be a single tensor in extreme cases where it was not wrapped properly but is [1,0].
            scale_rows, scale_cols = 1, 1
        if rows % scale_rows or cols % scale_cols:
            raise ValueError(
                f"Weight shape ({rows}, {cols}) not divisible by scale grid ({scale_rows}, {scale_cols})."
            )
        block_m = rows // scale_rows
        block_n = cols // scale_cols
        # ``ue8m0`` (``float8_e8m0fnu``) scales have no CUDA ``mul`` kernel, and casting
        # the FP8 weight to that dtype loses precision. Promote both sides to fp32 for
        # the math; prefer the destination parameter's dtype when known so eager modules
        # (e.g. plain ``nn.Linear``) keep the model's compute dtype after load.
        if output_dtype is None:
            output_dtype = (
                scales.dtype if scales.dtype.is_floating_point and scales.element_size() >= 2 else torch.bfloat16
            )
        # MXFP8 checkpoints ship E8M0 exponents stored as ``torch.uint8`` (one byte per
        # block) — the actual scale is `2 ** (byte - 127)`. Interpreting the raw bytes
        # as scalar multipliers would be silently wrong, so unpack to fp32 here.
        if scales.dtype == torch.uint8:
            s_fp32 = (scales.to(torch.float32) - 127.0).exp2()
        else:

View on GitHub (pinned to a597f97485)

Solutions

Verify pairing: print weight.shape[-2:] and scales.shape[-2:] for the failing key and confirm scales = ceil(rows/block_m) x ceil(cols/block_n)
If the scale was squeezed to a scalar/1x1 by mistake, restore its original [rows//bm, cols//bn] shape before dequantizing
Re-generate or re-download the checkpoint's weight_scale_inv tensors if they belong to a different block-size recipe
For custom checkpoints, regenerate scales with the block size your loader expects (e.g. 128x128 for FP8 dense, 1x32 for MXFP4 experts)

Example fix

# before
scale = scale_tensor.squeeze()  # accidentally [1,1] or 0-dim
deq = dequantize(weight, scale)
# ValueError: Weight shape (4096, 11008) not divisible by scale grid (1, 1)

# after
import math
scale = scale_tensor.reshape(weight.shape[0] // 128, weight.shape[1] // 128)
deq = dequantize(weight, scale)

Defensive patterns

Strategy: validation

Validate before calling

def check_scale_grid(weight, scales):
    rows, cols = weight.shape[-2:]
    sr, sc = scales.shape[-2:] if scales.dim() >= 2 else (1, 1)
    assert rows % sr == 0 and cols % sc == 0, (
        f"weight {rows}x{cols} not divisible by scale grid {sr}x{sc}"
    )

check_scale_grid(weight, scales)

Prevention

Never squeeze/flatten scale tensors when preprocessing checkpoints — preserve their grid shape
After resharding or merging checkpoints, assert weights.shape[-2:] tiles exactly by scales.shape[-2:] per key
Remember block sizes differ within one checkpoint: [1,32] for MXFP4 experts vs [128,128] for FP8 dense

When it happens

Trigger: Passing mismatched (weight, scale) pairs to the dequant helper — e.g. weight_scale_inv tensors from a different layer, a checkpoint saved with a different block size than the loader assumes, or a scale tensor collapsed to a scalar by improper wrapping (which hits the try/except and yields a 1x1 grid that rarely divides the weight).

Common situations: Loading a mixed-precision checkpoint (MXFP4 experts + FP8 dense) with a hand-edited or sharded state dict where scales were remapped; TP/shard resharding bugs that shuffle scale tensors; converting a checkpoint between block sizes without regenerating scales.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/12ab4f802a9de7b0. Report an issue: GitHub.