sgl-project/sglang · error · ValueError

Expected scalar scale for fused-in-checkpoint merged-column

Error message

Expected scalar scale for fused-in-checkpoint merged-column checkpoint load, got shape {tuple(loaded_weight.shape)}

What it means

In weight_loader_v2, when a fused-in-checkpoint shard (loaded_shard_id None or a tuple) is loaded into a PerTensorScaleParameter, the scale must be a single scalar (numel == 1). A non-scalar tensor of the fused width means the checkpoint stores per-shard/per-channel scales, incompatible with a per-tensor scale parameter.

Source

Thrown at python/sglang/srt/layers/linear.py:872

                loaded_weight=loaded_weight_shard,
                shard_id=shard_id,
                shard_offset=rank_shard_offset,
                shard_size=rank_shard_size,
                tp_rank=self.tp_rank,
                tp_size=self.tp_size,
                use_presharded_weights=self.use_presharded_weights,
            )

    def weight_loader_v2(
        self,
        param: BasevLLMParameter,
        loaded_weight: torch.Tensor,
        loaded_shard_id: tuple[int, ...] | int | None = None,
    ):
        if loaded_shard_id is None or isinstance(loaded_shard_id, tuple):
            if isinstance(param, PerTensorScaleParameter):
                if loaded_weight.numel() != 1:
                    raise ValueError(
                        "Expected scalar scale for fused-in-checkpoint "
                        "merged-column checkpoint load, got shape "
                        f"{tuple(loaded_weight.shape)}"
                    )
                if loaded_shard_id is None:
                    # The checkpoint tensor is already fused-in-checkpoint, so a
                    # scalar scale applies to the entire merged matrix. Fill
                    # every logical slot so later reductions only see valid
                    # scale values.
                    shard_ids = range(param.data.shape[0])
                else:
                    shard_ids = loaded_shard_id

                for shard_id in shard_ids:
                    param.load_merged_column_weight(
                        loaded_weight=loaded_weight,
                        shard_id=shard_id,
                        tp_rank=self.tp_rank,

View on GitHub (pinned to 0132848349)

Solutions

Align quant config: load the model with per-channel/per-block quant so the parameter is not PerTensorScaleParameter
Re-quantize/export the checkpoint with true per-tensor scales (single scalar per fused tensor)
If the checkpoint is genuinely fused with uniform scale, fix the exporter to store one scalar instead of a vector

Example fix

# before
# model loaded with quantization='fp8' (per-tensor) from a per-channel checkpoint
# loaded_weight.shape == (16384,) -> ValueError

# after
# load with a per-channel-capable quant config, or export checkpoint with scalar scale
# loaded_weight.numel() == 1

Defensive patterns

Strategy: validation

Validate before calling

if isinstance(param, PerTensorScaleParameter) and loaded_shard_id is not None:
    assert loaded_weight.numel() == 1, (
        f"per-tensor scale must be scalar, got shape {tuple(loaded_weight.shape)}; "
        "checkpoint/config quant recipe mismatch")

Type guard

def is_scalar_scale_for_fused_load(param, w, sid) -> bool:
    from sglang.srt.layers.quantization.fp8_utils import PerTensorScaleParameter
    if not isinstance(param, PerTensorScaleParameter):
        return True
    return sid is None or isinstance(sid, tuple) and w.numel() == 1

Try / catch

try:
    linear.weight_loader_v2(param, loaded_weight, loaded_shard_id)
except ValueError as e:
    if "Expected scalar scale" in str(e):
        raise RuntimeError("switch to per-channel quant config or re-export scalar scales") from e
    raise

Prevention

Keep checkpoint quant recipe and model quant config in sync (per-tensor vs per-channel)
Conversion scripts should emit exactly one scalar per fused per-tensor scale
Validate scale numel before load in checkpoint preprocessing

When it happens

Trigger: Loading a fused (e.g. gate_up or qkv-fused-in-checkpoint) weight's scale into a PerTensorScaleParameter where the checkpoint scale has more than one element (shape like (2*fan_in,) or (n_shards,)).

Common situations: Checkpoint quantized per-channel/per-group but model configured for per-tensor quantization; fused checkpoints written by converters that keep per-shard scales; config mix-ups between per-tensor and per-channel recipes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/129d60f3f6e5f4e0. Report an issue: GitHub.