sgl-project/sglang · error · ValueError

Unsupported activation: {config.hidden_act}. Only silu is su

Error message

Unsupported activation: {config.hidden_act}. Only silu is supported for now.

What it means

The MoE block, like the dense MLP, only supports SiluAndMul and rejects any config.hidden_act other than 'silu' at init.

Source

Thrown at python/sglang/srt/models/mimo_v2.py:397

        layer_id: int,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
        is_nextn: bool = False,
    ):
        super().__init__()
        self.tp_size = get_parallel().tp_size

        self.config = config
        self.layer_id = layer_id

        if self.tp_size > config.n_routed_experts:
            raise ValueError(
                f"Tensor parallel size {self.tp_size} is greater than "
                f"the number of experts {config.n_routed_experts}."
            )

        if config.hidden_act != "silu":
            raise ValueError(
                f"Unsupported activation: {config.hidden_act}. "
                "Only silu is supported for now."
            )

        self.gate = MoEGate(
            config=config,
            quant_config=quant_config,
            prefix=add_prefix("gate", prefix),
            is_nextn=is_nextn,
        )

        experts_type = get_moe_impl_class(quant_config)
        self.experts = experts_type(
            num_experts=config.n_routed_experts
            + get_exec().moe.ep_num_redundant_experts,
            top_k=config.num_experts_per_tok,
            hidden_size=config.hidden_size,
            intermediate_size=config.moe_intermediate_size,

View on GitHub (pinned to 0132848349)

Solutions

Set "hidden_act": "silu" in config.json
Extend the MoE block to support the needed activation if the checkpoint truly uses it
Use the model class matching the checkpoint architecture

Example fix

// before
"hidden_act": "gelu"
// after
"hidden_act": "silu"

Defensive patterns

Strategy: validation

Validate before calling

if config.hidden_act != 'silu':
    raise SystemExit('MiMo-v2 MoE requires hidden_act=silu')

Type guard

def moe_act_supported(cfg) -> bool:
    return getattr(cfg, 'hidden_act', None) == 'silu'

Prevention

Validate config.json activation field before model init
Don't reuse mimo_v2 classes for models with different MoE activations

When it happens

Trigger: Constructing the MiMo-v2 MoE layer with a config whose hidden_act != 'silu'.

Common situations: Modified config.json on fine-tuned MoE variants; derivative models reusing the mimo_v2 architecture with a different expert activation.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/90b2a130e4f4224f. Report an issue: GitHub.