{"record":{"id":"7971682068b9902b","repo":"huggingface/transformers","slug":"fp8dequantize-weight-scale-count-mismatch-for-ke","errorCode":null,"errorMessage":"Fp8Dequantize: weight/scale count mismatch for {key} ({len(weights)} weights vs {len(scales)} scales).","messagePattern":"Fp8Dequantize: weight/scale count mismatch for (.+?) \\((.+?) weights vs (.+?) scales\\)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/finegrained_fp8.py","lineNumber":1096,"sourceCode":"                scales = scales[0] if isinstance(scales, list) else scales\n                return {target_key: self._dequantize_one(quantized, scales, output_dtype=output_dtype)}\n            return {target_key: quantized}\n\n        # Generic chain path: dequantize every weight pattern that has a sibling scale.\n        result: dict[str, list[torch.Tensor] | torch.Tensor] = {}\n        for key, value in input_dict.items():\n            if \"activation_scale\" in key or \"weight_scale_inv\" in key:\n                continue  # consumed by the dequant; drop from the chain\n            scale_key = self._scale_pattern_for(key)\n            if scale_key not in input_dict:\n                # No scale to apply (e.g. unrelated entry) — pass through untouched.\n                result[key] = value\n                continue\n            weights = value if isinstance(value, list) else [value]\n            scales = input_dict[scale_key]\n            scales = scales if isinstance(scales, list) else [scales]\n            if len(weights) != len(scales):\n                raise ValueError(\n                    f\"Fp8Dequantize: weight/scale count mismatch for {key} \"\n                    f\"({len(weights)} weights vs {len(scales)} scales).\"\n                )\n            result[key] = [self._dequantize_one(w, s, output_dtype=output_dtype) for w, s in zip(weights, scales)]\n        return result\n\n    @property\n    def reverse_op(self) -> ConversionOps:\n        # Round-trip: dequantize on load -> re-quantize on save, so the saved\n        # checkpoint preserves the FP8 format (weight + per-block ``weight_scale_inv``)\n        # whether the in-memory state stayed quantized or was dequantized for compute.\n        return Fp8Quantize(self.hf_quantizer)\n","sourceCodeStart":1078,"sourceCodeEnd":1109,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/finegrained_fp8.py#L1078-L1109","documentation":"Fp8Dequantize is a tensor-conversion op that walks a state-dict fragment: for each weight key it finds the matching scale key (via _scale_pattern_for) and, when either side is a list (e.g. the 16 expert tensors of an MoE layer), the list lengths must match one-for-one. A mismatch means the checkpoint has a different number of weight shards than scale shards for that key — a corrupted or inconsistent checkpoint — and the zip would silently drop items, so a ValueError is raised instead.","triggerScenarios":"Loading a finegrained-fp8 MoE checkpoint where, for some key, len(weights) != len(scales): e.g. num_local_experts weight tensors but a different count of weight_scale_inv tensors because of sharding/resharding or a partial upload.","commonSituations":"Resharded or manually merged expert checkpoints (safetensors shard merge that lost some scale tensors); a quantized model saved before all experts had scales materialized; mismatched num_experts between the config and the tensors in the checkpoint.","solutions":["Inspect the failing key: print len of the weight list and the scale list in the state dict shard to see which experts are missing scales","Re-download the checkpoint (partial/interrupted downloads commonly produce missing tensors)","If you resharded or merged shards yourself, redo the merge ensuring weight_scale_inv entries are kept 1:1 with weights per expert","Verify the model config's num_local_experts matches the checkpoint's expert tensor count"],"exampleFix":"# before (mismatched expert counts after manual merge)\n# weights: 16 expert tensors, scales: 8 scale tensors\nstate = Fp8Dequantize(quantizer)(state_dict)  # ValueError\n\n# after: keep expert tensors and scales 1:1 when merging\nassert len(sd[\"mlp.experts.0.weight\"]) == len(sd[\"mlp.experts.0.weight_scale_inv\"])","handlingStrategy":"validation","validationCode":"def check_weight_scale_counts(sd):\n    for k, v in sd.items():\n        if \"weight_scale_inv\" in k or \"activation_scale\" in k:\n            continue\n        w = sd.get(k); s = sd.get(k.replace(\".weight\", \".weight_scale_inv\"))\n        if w is None or s is None:\n            continue\n        wn = len(w) if isinstance(w, list) else 1\n        sn = len(s) if isinstance(s, list) else 1\n        assert wn == sn, f\"{k}: {wn} weights vs {sn} scales\"\n\ncheck_weight_scale_counts(state_dict)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate expert weight/scale list lengths before loading sharded fp8 MoE checkpoints","Verify checkpoint downloads completed (file sizes vs index) before conversion","When merging safetensors shards, keep expert weight and weight_scale_inv entries paired 1:1"],"tags":["fp8","moe","checkpoint-corruption","state-dict","dequantization"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}