{"record":{"id":"129d60f3f6e5f4e0","repo":"sgl-project/sglang","slug":"expected-scalar-scale-for-fused-in-checkpoint-merg","errorCode":null,"errorMessage":"Expected scalar scale for fused-in-checkpoint merged-column checkpoint load, got shape {tuple(loaded_weight.shape)}","messagePattern":"Expected scalar scale for fused-in-checkpoint merged-column checkpoint load, got shape (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/linear.py","lineNumber":872,"sourceCode":"                loaded_weight=loaded_weight_shard,\n                shard_id=shard_id,\n                shard_offset=rank_shard_offset,\n                shard_size=rank_shard_size,\n                tp_rank=self.tp_rank,\n                tp_size=self.tp_size,\n                use_presharded_weights=self.use_presharded_weights,\n            )\n\n    def weight_loader_v2(\n        self,\n        param: BasevLLMParameter,\n        loaded_weight: torch.Tensor,\n        loaded_shard_id: tuple[int, ...] | int | None = None,\n    ):\n        if loaded_shard_id is None or isinstance(loaded_shard_id, tuple):\n            if isinstance(param, PerTensorScaleParameter):\n                if loaded_weight.numel() != 1:\n                    raise ValueError(\n                        \"Expected scalar scale for fused-in-checkpoint \"\n                        \"merged-column checkpoint load, got shape \"\n                        f\"{tuple(loaded_weight.shape)}\"\n                    )\n                if loaded_shard_id is None:\n                    # The checkpoint tensor is already fused-in-checkpoint, so a\n                    # scalar scale applies to the entire merged matrix. Fill\n                    # every logical slot so later reductions only see valid\n                    # scale values.\n                    shard_ids = range(param.data.shape[0])\n                else:\n                    shard_ids = loaded_shard_id\n\n                for shard_id in shard_ids:\n                    param.load_merged_column_weight(\n                        loaded_weight=loaded_weight,\n                        shard_id=shard_id,\n                        tp_rank=self.tp_rank,","sourceCodeStart":854,"sourceCodeEnd":890,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/linear.py#L854-L890","documentation":"In weight_loader_v2, when a fused-in-checkpoint shard (loaded_shard_id None or a tuple) is loaded into a PerTensorScaleParameter, the scale must be a single scalar (numel == 1). A non-scalar tensor of the fused width means the checkpoint stores per-shard/per-channel scales, incompatible with a per-tensor scale parameter.","triggerScenarios":"Loading a fused (e.g. gate_up or qkv-fused-in-checkpoint) weight's scale into a PerTensorScaleParameter where the checkpoint scale has more than one element (shape like (2*fan_in,) or (n_shards,)).","commonSituations":"Checkpoint quantized per-channel/per-group but model configured for per-tensor quantization; fused checkpoints written by converters that keep per-shard scales; config mix-ups between per-tensor and per-channel recipes.","solutions":["Align quant config: load the model with per-channel/per-block quant so the parameter is not PerTensorScaleParameter","Re-quantize/export the checkpoint with true per-tensor scales (single scalar per fused tensor)","If the checkpoint is genuinely fused with uniform scale, fix the exporter to store one scalar instead of a vector"],"exampleFix":"# before\n# model loaded with quantization='fp8' (per-tensor) from a per-channel checkpoint\n# loaded_weight.shape == (16384,) -> ValueError\n\n# after\n# load with a per-channel-capable quant config, or export checkpoint with scalar scale\n# loaded_weight.numel() == 1","handlingStrategy":"validation","validationCode":"if isinstance(param, PerTensorScaleParameter) and loaded_shard_id is not None:\n    assert loaded_weight.numel() == 1, (\n        f\"per-tensor scale must be scalar, got shape {tuple(loaded_weight.shape)}; \"\n        \"checkpoint/config quant recipe mismatch\")","typeGuard":"def is_scalar_scale_for_fused_load(param, w, sid) -> bool:\n    from sglang.srt.layers.quantization.fp8_utils import PerTensorScaleParameter\n    if not isinstance(param, PerTensorScaleParameter):\n        return True\n    return sid is None or isinstance(sid, tuple) and w.numel() == 1","tryCatchPattern":"try:\n    linear.weight_loader_v2(param, loaded_weight, loaded_shard_id)\nexcept ValueError as e:\n    if \"Expected scalar scale\" in str(e):\n        raise RuntimeError(\"switch to per-channel quant config or re-export scalar scales\") from e\n    raise","preventionTips":["Keep checkpoint quant recipe and model quant config in sync (per-tensor vs per-channel)","Conversion scripts should emit exactly one scalar per fused per-tensor scale","Validate scale numel before load in checkpoint preprocessing"],"tags":["weight-loading","per-tensor-scale","quantization","fused-checkpoint","shard-id"],"backgroundTag":"quantization-scale-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}