sgl-project/sglang · error · ValueError

Inkling shared-sink LoRA rank dimensions do not match

Error message

Inkling shared-sink LoRA rank dimensions do not match

What it means

set_lora_info derives max_rank from gate_up_lora_b.shape[-1] and requires the rank dims to be consistent: gate_up A's rank axis must be 2*max_rank (fused gate+up), down A's rank axis must be max_rank, and down B's rank axis must be max_rank. Violations mean the A/B rank pairs don't form valid low-rank factorizations.

Source

Thrown at python/sglang/srt/models/inkling_common/lora.py:76

            )
        if gate_outer != down_outer:
            raise ValueError(
                "Inkling shared-sink gate-up A and down B must use the same "
                "expert layout"
            )
        if (
            gate_up_lora_b_weights.shape[1] != self.n_shared_experts
            or down_lora_a_weights.shape[1] != self.n_shared_experts
        ):
            raise ValueError("Inkling shared-sink LoRA expert count does not match")

        max_rank = gate_up_lora_b_weights.shape[-1]
        if (
            gate_up_lora_a_weights.shape[2] != 2 * max_rank
            or down_lora_a_weights.shape[2] != max_rank
            or down_lora_b_weights.shape[-1] != max_rank
        ):
            raise ValueError("Inkling shared-sink LoRA rank dimensions do not match")

        self.set_lora = True
        self.gate_up_lora_a_weights = gate_up_lora_a_weights
        self.gate_up_lora_b_weights = gate_up_lora_b_weights
        self.down_lora_a_weights = down_lora_a_weights
        self.down_lora_b_weights = down_lora_b_weights
        self.experts_shared_outer_loras = gate_outer == 1
        self._allocate_lora_operands()
        self._refresh_lora_operands()

    def _allocate_lora_operands(self) -> None:
        if not self.experts_shared_outer_loras:
            self._w1_delta = None
            self._a_cat = None
            return
        slots, n, two_f, rank = self.gate_up_lora_b_weights.shape
        _, _, _, f = self.down_lora_a_weights.shape
        expected_gate_up = (slots, n * two_f, 2 * rank)

View on GitHub (pinned to 0132848349)

Solutions

1. Fuse gate and up A matrices along the rank axis to size 2*rank before binding
2. Make all four buffers use one rank r: A pairs (2r, r), B pairs (r, r)
3. Verify shapes programmatically: a.shape[rank_axis] == (2*)b.shape[-1] for each pair

Example fix

# before
gate_up_a = torch.cat([gate_a, up_a], dim=0)   # wrong axis -> rank mismatch
# after
gate_up_a = torch.cat([gate_a, up_a], dim=2)   # rank axis -> 2*r as required

r = gb.shape[-1]
assert ga.shape[2] == 2 * r and da.shape[2] == r and db.shape[-1] == r
module.set_lora_info(ga, gb, da, db)

def ranks_consistent(ga, gb, da, db) -> bool:
    r = gb.shape[-1]
    return ga.shape[2] == 2 * r and da.shape[2] == r and db.shape[-1] == r

When it happens

Trigger: Calling set_lora_info where gate_up_lora_a.shape[2] != 2 * gate_up_lora_b.shape[-1], or down_lora_a.shape[2] / down_lora_b.shape[-1] differ from max_rank — e.g. gate and up adapters not fused, or adapters of different ranks mixed into one bundle.

Common situations: Supplying separate gate and up LoRA matrices instead of the required fused (2r) form; mixing rank-16 and rank-32 adapters in one multi-slot pool; transposition mistakes putting rank on the wrong axis.

Related errors

• Inkling shared-sink LoRA requires four 4D MoE buffers
• Inkling shared-sink LoRA outer factors must have expert dime
• Inkling shared-sink gate-up A and down B must use the same e
• Inkling shared-sink LoRA expert count does not match
• Shared-sink LoRA pool shape changed after initialization: ga

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b2a50ede558e0fa0. Report an issue: GitHub.