sgl-project/sglang · error · ValueError

qkv_proj weight {name}: unexpected shape {tuple(loaded_weigh

Error message

qkv_proj weight {name}: unexpected shape {tuple(loaded_weight.shape)}; expected fused {fused_shape} or sharded {tuple(param.shape)}

What it means

When runtime tp_size equals the checkpoint tp, the loaded fused qkv weight must be exactly param.shape[0]*tp_size rows. This error means the fused tensor row count does not match tp_size * per-rank shard, so it can neither be chunked nor used as-is.

Source

Thrown at python/sglang/srt/models/mimo_v2.py:137

        if deferred_scale_inv is not None:
            deferred_scale_inv[name] = loaded_weight.clone()
            return
        raise ValueError(
            f"qkv_proj scale_inv {name}: shape mismatch "
            f"{tuple(loaded_weight.shape)} vs {tuple(param.shape)} "
            f"due to block quantization ceiling; pass deferred_scale_inv dict"
        )

    if loaded_weight.ndim != param.ndim or loaded_weight.shape[1:] != param.shape[1:]:
        raise ValueError(
            f"qkv_proj weight {name}: unexpected shape {tuple(loaded_weight.shape)}; "
            f"expected sharded {tuple(param.shape)}"
        )

    if tp_size == ckpt_tp:
        fused_shape = (param.shape[0] * tp_size, *param.shape[1:])
        if tuple(loaded_weight.shape) != fused_shape:
            raise ValueError(
                f"qkv_proj weight {name}: unexpected shape "
                f"{tuple(loaded_weight.shape)}; expected fused {fused_shape} "
                f"or sharded {tuple(param.shape)}"
            )
        default_weight_loader(param, loaded_weight.chunk(tp_size, dim=0)[tp_rank])
    else:
        shards_per_rank = ckpt_tp // tp_size
        shards = loaded_weight.chunk(ckpt_tp, dim=0)
        merged = torch.cat(
            shards[tp_rank * shards_per_rank : (tp_rank + 1) * shards_per_rank],
            dim=0,
        )
        default_weight_loader(param, merged)


def _get_ckpt_qkv_shard_sizes(config, layer_name, ckpt_tp):
    m = re.search(r"layers\.(\d+)\.", layer_name)
    if m is None:

View on GitHub (pinned to 0132848349)

Solutions

Check num_attention_heads and num_key_value_heads in config match the checkpoint's fused qkv shape (out = hidden + 2*kv_dim)
Re-run with a tp_size that divides the number of KV heads, or fix the config
Re-export the checkpoint with correct fused qkv ordering (interleaved q/k/v)

Defensive patterns

Strategy: validation

Validate before calling

c = config
kv = c.num_key_value_heads
expected_rows = c.hidden_size + 2 * kv * c.head_dim
if tuple(w.shape)[0] != expected_rows * tp_size // tp_size and tp == ckpt_tp:
    raise SystemExit('qkv fused shape mismatch; check num_kv_heads')

Type guard

def fused_qkv_ok(w: torch.Tensor, param: torch.Tensor, tp: int) -> bool:
    return tuple(w.shape) == (param.shape[0] * tp, *param.shape[1:])

Prevention

Ensure num_key_value_heads is divisible by tp_size
Pre-validate checkpoint qkv tensor shapes against model config

When it happens

Trigger: tp_size == ckpt_tp but tuple(loaded_weight.shape) != (param.shape[0]*tp_size, *param.shape[1:]) — e.g. num_kv_heads handling differs between checkpoint and config, producing a wrong fused row count (hidden + 2*kv rows).

Common situations: Mismatched num_key_value_heads between checkpoint and config, partially converted checkpoints, or GQA checkpoints where kv head count doesn't divide tp_size the same way.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8fb29724af113cd2. Report an issue: GitHub.