sgl-project/sglang · error · ValueError

{loaded_weight} are not all equal

Error message

{loaded_weight} are not all equal

What it means

On NPU (_is_npu), when a per-tensor quant scale parameter has size (1, ...) while the checkpoint scale has a larger leading dimension, the loader requires all rows of loaded_weight to be equal so it can safely collapse to one row; torch.allclose failing means the checkpoint stores non-uniform scales that don't fit a per-tensor parameter.

Source

Thrown at python/sglang/srt/layers/linear.py:284

        # (such scales for AutoFp8).
        if len(loaded_weight.shape) == 0:
            loaded_weight = loaded_weight.reshape(1)

        is_gguf_weight = getattr(param, "is_gguf_weight", False)
        is_gguf_weight_type = getattr(param, "is_gguf_weight_type", False)
        if is_gguf_weight_type:
            param.weight_type = loaded_weight.item()

        if is_gguf_weight and isinstance(param, UninitializedParameter):
            param.materialize(tuple(loaded_weight.shape), dtype=loaded_weight.dtype)

        # The per-tensor quant-scale must be 1 dimension
        if _is_npu:
            if param.size() != loaded_weight.size() and param.size(0) == 1:
                if torch.allclose(loaded_weight, loaded_weight[0]):
                    loaded_weight = loaded_weight[:1]
                else:
                    raise ValueError(f"{loaded_weight} are not all equal")

            if param.dtype == torch.int8 or loaded_weight.dtype == torch.int8:
                assert (
                    param.dtype == loaded_weight.dtype
                ), "init para dtype and loaded weight dtype should be the same"

        assert (
            param.size() == loaded_weight.size()
        ), f"{param.shape=} {param.dtype=} {loaded_weight.shape=} {loaded_weight.dtype=}"
        param.data.copy_(loaded_weight)

    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        bias = self.bias if not self.skip_bias_add else None
        assert self.quant_method is not None
        output = self.quant_method.apply(self, x, bias)
        output_bias = self.bias if self.skip_bias_add else None
        return output, output_bias

View on GitHub (pinned to 0132848349)

Solutions

1. Match the quant config to the checkpoint (use per-channel/per-block quant instead of per-tensor, or re-quantize the checkpoint to per-tensor so scales are uniform)
2. Verify the checkpoint's scale tensor really should be uniform; if it's per-TP-shard, load the correct rank's slice
3. If scales legitimately differ, resize/redesign the param to hold per-channel scales rather than 1 row

Example fix

# before
# quant_config describes per-tensor but checkpoint has per-channel scales
loader(param=torch.Size([1, N]), loaded_weight=torch.Size([8, N]))  # raises

# after
# re-quantize checkpoint per-tensor, or configure per-channel quant so param is [8, N]

if _is_npu and param.size(0) == 1 and loaded_weight.size(0) > 1:
    assert loaded_weight.unique(dim=0).size(0) == 1, (
        "checkpoint scales are non-uniform; per-tensor param cannot hold them")

try:
    linear.weight_loader(param, loaded_weight, loaded_shard_id)
except ValueError as e:
    if "are not all equal" in str(e):
        raise RuntimeError("per-channel checkpoint vs per-tensor quant config mismatch on NPU") from e
    raise

When it happens

Trigger: Loading a checkpoint whose quant scale tensor has shape [N, ...] with non-identical entries into a param of size [1, ...] on NPU hardware — e.g. a per-channel (or per-shard) scale loaded into a layer quantized per-tensor.

Common situations: Mixing a per-channel quantized checkpoint with a per-tensor quant config on NPU; TP shards carrying different scales being concatenated; checkpoint conversion scripts emitting duplicated-but-diverged scale rows.

Related errors

• Unknown Shard Id {shard_id}
• expected a tensor with at least one dimension
• dimension {dim} size {dim_size} must be divisible by 2 * gro
• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e97d7880e0704674. Report an issue: GitHub.