{"record":{"id":"f2a33799bb36efc6","repo":"sgl-project/sglang","slug":"expected-a-3d-packed-tensor-for-name-got-loade","errorCode":null,"errorMessage":"Expected a 3D packed tensor for {name}, got {loaded_weight.dim()}D {tuple(loaded_weight.shape)}","messagePattern":"Expected a 3D packed tensor for (.+?), got (.+?)D (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/models/lfm2_moe.py","lineNumber":617,"sourceCode":"            # per projection (experts.gate_up_proj / experts.down_proj) instead of\n            # per-expert weights (experts.{i}.w{1,2,3}.weight). This is the layout an\n            # in-memory Transformers model exposes -- e.g. the update_weights_from_tensor\n            # / RLHF weight-sync path -- so map the packed tensors onto the fused\n            # FusedMoE params (w13_weight / w2_weight) per expert. LFM2-MoE packs\n            # out-features-major (gate_up_proj as [num_experts, 2 * intermediate,\n            # hidden], down_proj as [num_experts, hidden, intermediate]), matching the\n            # FusedMoE layout, so no transpose is needed.\n            if \"feed_forward.experts.gate_up_proj\" in name:\n                fused_name = name\n                if fused_name.endswith(\".weight\"):\n                    fused_name = fused_name[: -len(\".weight\")]\n                fused_name = fused_name.replace(\n                    \"feed_forward.experts.gate_up_proj\",\n                    \"feed_forward.experts.w13_weight\",\n                )\n                if fused_name in params_dict:\n                    if loaded_weight.dim() != 3:\n                        raise ValueError(\n                            f\"Expected a 3D packed tensor for {name}, got \"\n                            f\"{loaded_weight.dim()}D {tuple(loaded_weight.shape)}\"\n                        )\n                    param = params_dict[fused_name]\n                    weight_loader = param.weight_loader\n                    if loaded_weight.shape[1] % 2 != 0:\n                        raise ValueError(\n                            f\"Invalid gate_up_proj shape for {name}: \"\n                            f\"{tuple(loaded_weight.shape)}\"\n                        )\n                    w1, w3 = loaded_weight.chunk(2, dim=1)\n                    for expert_id in range(w1.shape[0]):\n                        weight_loader(\n                            param,\n                            w1[expert_id],\n                            fused_name,\n                            shard_id=\"w1\",\n                            expert_id=expert_id,","sourceCodeStart":599,"sourceCodeEnd":635,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/lfm2_moe.py#L599-L635","documentation":"When remapping fused gate_up_proj expert weights to experts.w13_weight, LFM2-MoE's load_weights (lfm2_moe.py:617) requires a 3D [num_experts, 2*intermediate, hidden] packed tensor. A 1D/2D tensor means the checkpoint stores experts unfused or flattened differently than expected, and the chunk-along-dim-1 split into w1/w3 would corrupt weights.","triggerScenarios":"Loading a checkpoint whose model.experts.gate_up_proj.weight is 2D (single fused expert, not expert-packed) or 4D, into the fused-w13 loader path.","commonSituations":"Custom conversions, checkpoints from frameworks that don't pack experts into dim 0, or dense-FFN variants mislabeled as MoE.","solutions":["Re-convert/reshape the checkpoint to [num_experts, 2*intermediate_size, hidden_size]","Load with a converter that produces stacked expert tensors (e.g. SGLang/vLLM conversion scripts for LFM2)","Verify the checkpoint is actually the MoE LFM2 variant, not the dense one"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"w = load_tensor(path, \"...experts.gate_up_proj.weight\")\nassert w.dim() == 3, w.shape","typeGuard":"def is_packed_expert_tensor(t) -> bool:\n    return t.dim() == 3 and t.shape[0] == cfg.num_experts","tryCatchPattern":null,"preventionTips":["Run a shape-audit script over converted checkpoints before serving"],"tags":["lfm2","moe","weight-loading","tensor-shape"],"backgroundTag":"checkpoint-tensor-rank-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}