{"record":{"id":"2f83602954fa267e","repo":"sgl-project/sglang","slug":"shared-sink-down-lora-a-width-must-be-divisible-by","errorCode":null,"errorMessage":"Shared-sink down LoRA-A width must be divisible by {self.n_shared_experts}, got {flat_intermediate}","messagePattern":"Shared-sink down LoRA-A width must be divisible by (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/inkling_common/lora.py","lineNumber":145,"sourceCode":"            for slot in slot_ids:\n                gate_up[slot].zero_()\n                gate_up[slot, :, :, 0, :rank].copy_(b_gate_up[slot, :, :f, :])\n                gate_up[slot, :, :, 1, rank:].copy_(b_gate_up[slot, :, f:, :])\n                a_cat[slot].copy_(a_down[slot].permute(1, 0, 2))\n\n    def slice_moe_lora_a_weights(\n        self,\n        weights: torch.Tensor | dict[int, torch.Tensor],\n        tp_rank: int,\n        target_module: str,\n    ) -> torch.Tensor | dict[int, torch.Tensor]:\n        if isinstance(weights, torch.Tensor) and weights.dim() == 2:\n            if target_module == \"gate_up_proj_moe\":\n                weights = weights.unsqueeze(0)\n            else:\n                rank, flat_intermediate = weights.shape\n                if flat_intermediate % self.n_shared_experts != 0:\n                    raise ValueError(\n                        \"Shared-sink down LoRA-A width must be divisible by \"\n                        f\"{self.n_shared_experts}, got {flat_intermediate}\"\n                    )\n                weights = (\n                    weights.view(\n                        rank,\n                        self.n_shared_experts,\n                        flat_intermediate // self.n_shared_experts,\n                    )\n                    .transpose(0, 1)\n                    .contiguous()\n                )\n        if self.moe_tp_size <= 1 or target_module != \"down_proj_moe\":\n            return weights\n        if isinstance(weights, dict):\n            return {\n                expert_id: self._slice_down_lora_a(weight, tp_rank)\n                for expert_id, weight in weights.items()","sourceCodeStart":127,"sourceCodeEnd":163,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/inkling_common/lora.py#L127-L163","documentation":"slice_moe_lora_a_weights validates that a 2-D down-proj LoRA-A weight's second dimension (flat intermediate width) is divisible by n_shared_experts before reshaping it into per-shared-expert slices. A non-divisible width cannot be evenly split across shared experts, so the reshape is rejected.","triggerScenarios":"Loading a LoRA adapter whose down_proj LoRA-A matrix has flat_intermediate % n_shared_experts != 0 while target_module is not gate_up_proj_moe.","commonSituations":"Adapter checkpoint trained for a different shared-expert count or intermediate size than the serving model; exporting LoRA with a mismatched rank/width layout.","solutions":["Verify the adapter's LoRA-A width equals intermediate_size (which is divisible by n_shared_experts) of the target model","Retrain or re-export the LoRA adapter against the same model config (n_shared_experts and intermediate size)","Route gate_up modules through gate_up_proj_moe so they are unsqueezed instead of validated against n_shared_experts"],"exampleFix":"# before\nlora_a = torch.randn(rank, 3072)  # model has 2 shared experts, 3072 % 2 -> ok only if divisible; e.g. 3073 fails\n# after\nassert lora_a.shape[1] % model.n_shared_experts == 0\nlora_a = torch.randn(rank, model.intermediate_size)","handlingStrategy":"validation","validationCode":"if lora_a.dim() == 2 and lora_a.shape[1] % model.n_shared_experts != 0:\n    raise ValueError('bad LoRA-A width before load')","typeGuard":"def is_valid_lora_a(w, n_shared):\n    return w.dim() == 2 and w.shape[1] % n_shared == 0","tryCatchPattern":null,"preventionTips":["Validate adapter shapes against the target config before loading","Export adapters with the same intermediate size as the base model"],"tags":["lora","shape-mismatch","weight-loading","moe"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}