{"record":{"id":"e97d7880e0704674","repo":"sgl-project/sglang","slug":"loaded-weight-are-not-all-equal","errorCode":null,"errorMessage":"{loaded_weight} are not all equal","messagePattern":"(.+?) are not all equal","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/linear.py","lineNumber":284,"sourceCode":"        # (such scales for AutoFp8).\n        if len(loaded_weight.shape) == 0:\n            loaded_weight = loaded_weight.reshape(1)\n\n        is_gguf_weight = getattr(param, \"is_gguf_weight\", False)\n        is_gguf_weight_type = getattr(param, \"is_gguf_weight_type\", False)\n        if is_gguf_weight_type:\n            param.weight_type = loaded_weight.item()\n\n        if is_gguf_weight and isinstance(param, UninitializedParameter):\n            param.materialize(tuple(loaded_weight.shape), dtype=loaded_weight.dtype)\n\n        # The per-tensor quant-scale must be 1 dimension\n        if _is_npu:\n            if param.size() != loaded_weight.size() and param.size(0) == 1:\n                if torch.allclose(loaded_weight, loaded_weight[0]):\n                    loaded_weight = loaded_weight[:1]\n                else:\n                    raise ValueError(f\"{loaded_weight} are not all equal\")\n\n            if param.dtype == torch.int8 or loaded_weight.dtype == torch.int8:\n                assert (\n                    param.dtype == loaded_weight.dtype\n                ), \"init para dtype and loaded weight dtype should be the same\"\n\n        assert (\n            param.size() == loaded_weight.size()\n        ), f\"{param.shape=} {param.dtype=} {loaded_weight.shape=} {loaded_weight.dtype=}\"\n        param.data.copy_(loaded_weight)\n\n    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:\n        bias = self.bias if not self.skip_bias_add else None\n        assert self.quant_method is not None\n        output = self.quant_method.apply(self, x, bias)\n        output_bias = self.bias if self.skip_bias_add else None\n        return output, output_bias\n","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/linear.py#L266-L302","documentation":"On NPU (_is_npu), when a per-tensor quant scale parameter has size (1, ...) while the checkpoint scale has a larger leading dimension, the loader requires all rows of loaded_weight to be equal so it can safely collapse to one row; torch.allclose failing means the checkpoint stores non-uniform scales that don't fit a per-tensor parameter.","triggerScenarios":"Loading a checkpoint whose quant scale tensor has shape [N, ...] with non-identical entries into a param of size [1, ...] on NPU hardware — e.g. a per-channel (or per-shard) scale loaded into a layer quantized per-tensor.","commonSituations":"Mixing a per-channel quantized checkpoint with a per-tensor quant config on NPU; TP shards carrying different scales being concatenated; checkpoint conversion scripts emitting duplicated-but-diverged scale rows.","solutions":["Match the quant config to the checkpoint (use per-channel/per-block quant instead of per-tensor, or re-quantize the checkpoint to per-tensor so scales are uniform)","Verify the checkpoint's scale tensor really should be uniform; if it's per-TP-shard, load the correct rank's slice","If scales legitimately differ, resize/redesign the param to hold per-channel scales rather than 1 row"],"exampleFix":"# before\n# quant_config describes per-tensor but checkpoint has per-channel scales\nloader(param=torch.Size([1, N]), loaded_weight=torch.Size([8, N]))  # raises\n\n# after\n# re-quantize checkpoint per-tensor, or configure per-channel quant so param is [8, N]","handlingStrategy":"validation","validationCode":"if _is_npu and param.size(0) == 1 and loaded_weight.size(0) > 1:\n    assert loaded_weight.unique(dim=0).size(0) == 1, (\n        \"checkpoint scales are non-uniform; per-tensor param cannot hold them\")","typeGuard":null,"tryCatchPattern":"try:\n    linear.weight_loader(param, loaded_weight, loaded_shard_id)\nexcept ValueError as e:\n    if \"are not all equal\" in str(e):\n        raise RuntimeError(\"per-channel checkpoint vs per-tensor quant config mismatch on NPU\") from e\n    raise","preventionTips":["Match quant recipe between checkpoint export and load config","Validate scale uniformity when converting checkpoints for per-tensor NPU quant","Add a pre-load shape/consistency check for scale tensors"],"tags":["npu","quant-scale","weight-loading","per-tensor-quant","allclose"],"backgroundTag":"quantization-scale-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}