sgl-project/sglang · critical · ValueError

Invalid gate_up_proj shape for {name}: {tuple(loaded_weight.

Error message

Invalid gate_up_proj shape for {name}: {tuple(loaded_weight.shape)}

What it means

Before chunking the packed gate_up tensor into w1/w3 along dim 1, LFM2-MoE checks loaded_weight.shape[1] % 2 == 0 (lfm2_moe.py:624). An odd second dimension cannot be evenly split into gate and up projections, so the tensor is malformed for the fused layout and loading aborts.

Source

Thrown at python/sglang/srt/models/lfm2_moe.py:624

            # FusedMoE layout, so no transpose is needed.
            if "feed_forward.experts.gate_up_proj" in name:
                fused_name = name
                if fused_name.endswith(".weight"):
                    fused_name = fused_name[: -len(".weight")]
                fused_name = fused_name.replace(
                    "feed_forward.experts.gate_up_proj",
                    "feed_forward.experts.w13_weight",
                )
                if fused_name in params_dict:
                    if loaded_weight.dim() != 3:
                        raise ValueError(
                            f"Expected a 3D packed tensor for {name}, got "
                            f"{loaded_weight.dim()}D {tuple(loaded_weight.shape)}"
                        )
                    param = params_dict[fused_name]
                    weight_loader = param.weight_loader
                    if loaded_weight.shape[1] % 2 != 0:
                        raise ValueError(
                            f"Invalid gate_up_proj shape for {name}: "
                            f"{tuple(loaded_weight.shape)}"
                        )
                    w1, w3 = loaded_weight.chunk(2, dim=1)
                    for expert_id in range(w1.shape[0]):
                        weight_loader(
                            param,
                            w1[expert_id],
                            fused_name,
                            shard_id="w1",
                            expert_id=expert_id,
                        )
                        weight_loader(
                            param,
                            w3[expert_id],
                            fused_name,
                            shard_id="w3",
                            expert_id=expert_id,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-download / regenerate the checkpoint with correct fused shapes ([E, 2*I, H])
2. Validate shape[1] is even and equals 2*config.intermediate_size before loading
3. Report to the checkpoint publisher if the shipped file is malformed

assert w.dim() == 3 and w.shape[1] == 2 * cfg.intermediate_size and w.shape[1] % 2 == 0

When it happens

Trigger: A gate_up_proj checkpoint tensor with odd intermediate dimension (e.g. shape [E, 1101, H]) reaching the w13 loader - usually a botched conversion or truncated tensor.

Common situations: Manual safetensors surgery, quantization tooling that rounds intermediate sizes, corrupted downloads.

Related errors

• Expected a 3D packed tensor for {name}, got {loaded_weight.d
• expected a tensor with at least one dimension
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• Inkling shared-sink LoRA requires four 4D MoE buffers

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bfbf054f5266004b. Report an issue: GitHub.