sgl-project/sglang · error · ValueError

qkv_proj scale_inv {name}: shape mismatch {tuple(loaded_weig

Error message

qkv_proj scale_inv {name}: shape mismatch {tuple(loaded_weight.shape)} vs {tuple(param.shape)} due to block quantization ceiling; pass deferred_scale_inv dict

What it means

Raised while loading qkv_proj weight_scale_inv tensors for MiMo-v2 when the checkpoint's tensor-parallel degree differs from the runtime tp_size. Block-quantized scale_inv tensors cannot be naively resharded because their spatial blocks only cover a ceiling of rows, so the loader requires the caller to pass a deferred_scale_inv dict to collect them for later resolution.

Source

Thrown at python/sglang/srt/models/mimo_v2.py:122

    ckpt_tp = expected_fused_tp_size if expected_fused_tp_size is not None else tp_size

    if ckpt_tp == tp_size and loaded_weight.shape == param.shape:
        default_weight_loader(param, loaded_weight)
        return

    if expected_fused_tp_size is not None and expected_fused_tp_size % tp_size != 0:
        raise ValueError(
            f"MiMoV2 fused qkv_proj checkpoint is TP={expected_fused_tp_size}-"
            f"interleaved; got attention tp_size={tp_size} while loading {name}."
        )

    is_scale_inv = "weight_scale_inv" in name

    if is_scale_inv and ckpt_tp != tp_size:
        if deferred_scale_inv is not None:
            deferred_scale_inv[name] = loaded_weight.clone()
            return
        raise ValueError(
            f"qkv_proj scale_inv {name}: shape mismatch "
            f"{tuple(loaded_weight.shape)} vs {tuple(param.shape)} "
            f"due to block quantization ceiling; pass deferred_scale_inv dict"
        )

    if loaded_weight.ndim != param.ndim or loaded_weight.shape[1:] != param.shape[1:]:
        raise ValueError(
            f"qkv_proj weight {name}: unexpected shape {tuple(loaded_weight.shape)}; "
            f"expected sharded {tuple(param.shape)}"
        )

    if tp_size == ckpt_tp:
        fused_shape = (param.shape[0] * tp_size, *param.shape[1:])
        if tuple(loaded_weight.shape) != fused_shape:
            raise ValueError(
                f"qkv_proj weight {name}: unexpected shape "
                f"{tuple(loaded_weight.shape)}; expected fused {fused_shape} "
                f"or sharded {tuple(param.shape)}"

View on GitHub (pinned to 0132848349)

Solutions

1. Pass a dict as deferred_scale_inv when calling load_mimo_v2_qkv_proj_weight / load_weights so scale_inv tensors are collected and resolved later by _resolve_deferred_qkv_scale_inv
2. Ensure you use the standard model.load_weights path (ModelRunner) which already threads deferred_scale_inv through
3. Match runtime tp_size to the checkpoint's fused tp size so is_scale_inv resharding is skipped

Example fix

# before
model.load_weights(weights)  # scale_inv path raises

# after
deferred = {}
model.load_weights(weights, deferred_scale_inv=deferred)
_resolve_deferred_qkv_scale_inv(model, deferred, ...)

expected = getattr(loaded, 'tp_size', 1)
if 'weight_scale_inv' in name and expected != runtime_tp:
    deferred = deferred or {}
    deferred[name] = loaded
    return

def needs_deferred_scale_inv(name: str, ckpt_tp: int, tp: int) -> bool:
    return name.endswith('weight_scale_inv') and ckpt_tp != tp

When it happens

Trigger: Loading a block-quantized (e.g. AWQ/GPTQ-style weight_scale_inv) MiMo-v2 checkpoint with a tp_size different from ckpt_tp in load_mimo_v2_qkv_proj_weight without passing deferred_scale_inv to load_weights.

Common situations: Running tp=2 or tp=4 on a checkpoint saved for tp=1 (or vice versa) with block quantization; custom weight-loading code that calls the loader directly without the dict.

Related errors

• Cannot resolve deferred scale_inv {scale_name}: weight {weig
• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP does not support nested quant states.
• bitsandbytes 4-bit TP shard is not aligned to quantization b
• Weight input_size_per_partition = {input_size_per_partition}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9bbbca9f49773f9a. Report an issue: GitHub.