sgl-project/sglang · critical · ValueError

Expected a 3D packed tensor for {name}, got {loaded_weight.d

Error message

Expected a 3D packed tensor for {name}, got {loaded_weight.dim()}D {tuple(loaded_weight.shape)}

What it means

When remapping fused gate_up_proj expert weights to experts.w13_weight, LFM2-MoE's load_weights (lfm2_moe.py:617) requires a 3D [num_experts, 2*intermediate, hidden] packed tensor. A 1D/2D tensor means the checkpoint stores experts unfused or flattened differently than expected, and the chunk-along-dim-1 split into w1/w3 would corrupt weights.

Source

Thrown at python/sglang/srt/models/lfm2_moe.py:617

            # per projection (experts.gate_up_proj / experts.down_proj) instead of
            # per-expert weights (experts.{i}.w{1,2,3}.weight). This is the layout an
            # in-memory Transformers model exposes -- e.g. the update_weights_from_tensor
            # / RLHF weight-sync path -- so map the packed tensors onto the fused
            # FusedMoE params (w13_weight / w2_weight) per expert. LFM2-MoE packs
            # out-features-major (gate_up_proj as [num_experts, 2 * intermediate,
            # hidden], down_proj as [num_experts, hidden, intermediate]), matching the
            # FusedMoE layout, so no transpose is needed.
            if "feed_forward.experts.gate_up_proj" in name:
                fused_name = name
                if fused_name.endswith(".weight"):
                    fused_name = fused_name[: -len(".weight")]
                fused_name = fused_name.replace(
                    "feed_forward.experts.gate_up_proj",
                    "feed_forward.experts.w13_weight",
                )
                if fused_name in params_dict:
                    if loaded_weight.dim() != 3:
                        raise ValueError(
                            f"Expected a 3D packed tensor for {name}, got "
                            f"{loaded_weight.dim()}D {tuple(loaded_weight.shape)}"
                        )
                    param = params_dict[fused_name]
                    weight_loader = param.weight_loader
                    if loaded_weight.shape[1] % 2 != 0:
                        raise ValueError(
                            f"Invalid gate_up_proj shape for {name}: "
                            f"{tuple(loaded_weight.shape)}"
                        )
                    w1, w3 = loaded_weight.chunk(2, dim=1)
                    for expert_id in range(w1.shape[0]):
                        weight_loader(
                            param,
                            w1[expert_id],
                            fused_name,
                            shard_id="w1",
                            expert_id=expert_id,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-convert/reshape the checkpoint to [num_experts, 2*intermediate_size, hidden_size]
2. Load with a converter that produces stacked expert tensors (e.g. SGLang/vLLM conversion scripts for LFM2)
3. Verify the checkpoint is actually the MoE LFM2 variant, not the dense one

w = load_tensor(path, "...experts.gate_up_proj.weight")
assert w.dim() == 3, w.shape

def is_packed_expert_tensor(t) -> bool:
    return t.dim() == 3 and t.shape[0] == cfg.num_experts

When it happens

Trigger: Loading a checkpoint whose model.experts.gate_up_proj.weight is 2D (single fused expert, not expert-packed) or 4D, into the fused-w13 loader path.

Common situations: Custom conversions, checkpoints from frameworks that don't pack experts into dim 0, or dense-FFN variants mislabeled as MoE.

Related errors

• Invalid gate_up_proj shape for {name}: {tuple(loaded_weight.
• num_token_non_padded must be a torch.Tensor
• num_token_non_padded must be a single-element tensor, got sh
• Pack: Only supports tensors with dimensions not greater than
• Shared-sink down LoRA-A width must be divisible by {self.n_s

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f2a33799bb36efc6. Report an issue: GitHub.