sgl-project/sglang · error · ValueError

Expected {num_experts} experts in {name}, got {loaded_weight

Error message

Expected {num_experts} experts in {name}, got {loaded_weight.shape[0]}

What it means

When loading a fused Mobius gate/up expert tensor, its leading dimension (expert count in the checkpoint) must equal the model's num_experts; otherwise slicing per expert would be wrong and it raises ValueError.

Source

Thrown at python/sglang/srt/models/interns2_mobius.py:122

    record_slot,
) -> None:
    gate_up_suffixes = {
        "experts.gate_up_proj": "experts.w13_weight",
        "experts.gate_up_proj_scale_inv": "experts.w13_weight_scale_inv",
    }
    gate_up_suffix = next(
        (suffix for suffix in gate_up_suffixes if name.endswith(suffix)), None
    )
    if gate_up_suffix is not None:
        parameter_name = (
            name.removesuffix(gate_up_suffix) + gate_up_suffixes[gate_up_suffix]
        )
        if parameter_name not in params_dict:
            raise KeyError(
                f"Mobius fused gate/up destination is missing: {parameter_name}"
            )
        if loaded_weight.shape[0] != num_experts:
            raise ValueError(
                f"Expected {num_experts} experts in {name}, got {loaded_weight.shape[0]}"
            )
        gate_weights, up_weights = loaded_weight.chunk(2, dim=-2)
        parameter = params_dict[parameter_name]
        loader = parameter.weight_loader
        for expert_id in range(num_experts):
            for shard_id, expert_weight in (
                ("w1", gate_weights[expert_id]),
                ("w3", up_weights[expert_id]),
            ):
                record_slot(parameter_name, shard_id, expert_id)
                loader(
                    parameter,
                    expert_weight,
                    parameter_name,
                    shard_id,
                    expert_id,
                )

View on GitHub (pinned to 0132848349)

Solutions

1. Align the config's num_local/moe experts with the checkpoint's expert dimension
2. If using expert parallelism, load the correct local shard so shape[0] equals the per-rank expert count

Example fix

# before
# config.num_experts = 32, checkpoint tensor shape [64, N, K]
# after
config.num_experts = 64  # or shard the tensor for EP

assert loaded_weight.shape[0] == config.num_experts, (loaded_weight.shape, config.num_experts)

When it happens

Trigger: loaded_weight.shape[0] != num_experts for a tensor matching a gate_up suffix — e.g. checkpoint trained with a different expert count than the served config.

Common situations: Model config num_experts changed between training and serving; partial expert-parallel shard loaded without the EP dimension handled.

Related errors

• Shared-sink down LoRA-A width must be divisible by {self.n_s
• Shared-sink gate/up LoRA-B height must be divisible by {self
• Mobius fused gate/up destination is missing: {parameter_name
• {len(missing)} routed-expert tensors were not loaded (sample
• Expected a 3D packed tensor for {name}, got {loaded_weight.d

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5f1385b97768ebad. Report an issue: GitHub.