huggingface/transformers · error · ValueError

Fp8Dequantize: weight/scale count mismatch for {key} ({len(w

Error message

Fp8Dequantize: weight/scale count mismatch for {key} ({len(weights)} weights vs {len(scales)} scales).

What it means

Fp8Dequantize is a tensor-conversion op that walks a state-dict fragment: for each weight key it finds the matching scale key (via _scale_pattern_for) and, when either side is a list (e.g. the 16 expert tensors of an MoE layer), the list lengths must match one-for-one. A mismatch means the checkpoint has a different number of weight shards than scale shards for that key — a corrupted or inconsistent checkpoint — and the zip would silently drop items, so a ValueError is raised instead.

Source

Thrown at src/transformers/integrations/finegrained_fp8.py:1096

                scales = scales[0] if isinstance(scales, list) else scales
                return {target_key: self._dequantize_one(quantized, scales, output_dtype=output_dtype)}
            return {target_key: quantized}

        # Generic chain path: dequantize every weight pattern that has a sibling scale.
        result: dict[str, list[torch.Tensor] | torch.Tensor] = {}
        for key, value in input_dict.items():
            if "activation_scale" in key or "weight_scale_inv" in key:
                continue  # consumed by the dequant; drop from the chain
            scale_key = self._scale_pattern_for(key)
            if scale_key not in input_dict:
                # No scale to apply (e.g. unrelated entry) — pass through untouched.
                result[key] = value
                continue
            weights = value if isinstance(value, list) else [value]
            scales = input_dict[scale_key]
            scales = scales if isinstance(scales, list) else [scales]
            if len(weights) != len(scales):
                raise ValueError(
                    f"Fp8Dequantize: weight/scale count mismatch for {key} "
                    f"({len(weights)} weights vs {len(scales)} scales)."
                )
            result[key] = [self._dequantize_one(w, s, output_dtype=output_dtype) for w, s in zip(weights, scales)]
        return result

    @property
    def reverse_op(self) -> ConversionOps:
        # Round-trip: dequantize on load -> re-quantize on save, so the saved
        # checkpoint preserves the FP8 format (weight + per-block ``weight_scale_inv``)
        # whether the in-memory state stayed quantized or was dequantized for compute.
        return Fp8Quantize(self.hf_quantizer)

View on GitHub (pinned to a597f97485)

Solutions

1. Inspect the failing key: print len of the weight list and the scale list in the state dict shard to see which experts are missing scales
2. Re-download the checkpoint (partial/interrupted downloads commonly produce missing tensors)
3. If you resharded or merged shards yourself, redo the merge ensuring weight_scale_inv entries are kept 1:1 with weights per expert
4. Verify the model config's num_local_experts matches the checkpoint's expert tensor count

Example fix

# before (mismatched expert counts after manual merge)
# weights: 16 expert tensors, scales: 8 scale tensors
state = Fp8Dequantize(quantizer)(state_dict)  # ValueError

# after: keep expert tensors and scales 1:1 when merging
assert len(sd["mlp.experts.0.weight"]) == len(sd["mlp.experts.0.weight_scale_inv"])

def check_weight_scale_counts(sd):
    for k, v in sd.items():
        if "weight_scale_inv" in k or "activation_scale" in k:
            continue
        w = sd.get(k); s = sd.get(k.replace(".weight", ".weight_scale_inv"))
        if w is None or s is None:
            continue
        wn = len(w) if isinstance(w, list) else 1
        sn = len(s) if isinstance(s, list) else 1
        assert wn == sn, f"{k}: {wn} weights vs {sn} scales"

check_weight_scale_counts(state_dict)

When it happens

Trigger: Loading a finegrained-fp8 MoE checkpoint where, for some key, len(weights) != len(scales): e.g. num_local_experts weight tensors but a different count of weight_scale_inv tensors because of sharding/resharding or a partial upload.

Common situations: Resharded or manually merged expert checkpoints (safetensors shard merge that lost some scale tensors); a quantized model saved before all experts had scales materialized; mismatched num_experts between the config and the tensors in the checkpoint.

Related errors

• Incorrect audio source. Must be a valid URL starting with `h
• batched_mm experts dispatch does not support activation_sche
• grouped_mm experts dispatch does not support activation_sche
• Weight shape ({rows}, {cols}) not divisible by scale grid ({
• No benchmark was run successfully

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7971682068b9902b. Report an issue: GitHub.