What causes "Sparse MLP requested but num_experts <= 0 in Mellum config" in sgl-project/sglang?

Set num_experts to the checkpoint's expert count (e.g. 128) in config.json

sgl-project/sglang · critical · ValueError

Sparse MLP requested but num_experts <= 0 in Mellum config

Error message

Sparse MLP requested but num_experts <= 0 in Mellum config

What it means

When a Mellum layer is marked 'sparse' in mlp_layer_types, the model instantiates a Qwen3MoeSparseMoeBlock which requires config.num_experts > 0. If num_experts is missing (0 or negative), the sparse routing head cannot be built and init aborts.

Source

Thrown at python/sglang/srt/models/mellum.py:425

                f"{len(mlp_layer_types)} and {cfg.num_hidden_layers}"
            )

        def _is_sparse(lid: int) -> bool:
            if lid < 0 or lid >= cfg.num_hidden_layers:
                return False
            mlp_type = mlp_layer_types[lid]
            if mlp_type not in ("sparse", "dense"):
                raise ValueError(
                    f"Unsupported mlp_layer_types[{lid}]={mlp_type}; "
                    "expected 'sparse' or 'dense'"
                )
            return mlp_type == "sparse"

        self.is_layer_sparse = _is_sparse(layer_id)

        if self.is_layer_sparse:
            if num_experts <= 0:
                raise ValueError(
                    "Sparse MLP requested but num_experts <= 0 in Mellum config"
                )
            self.mlp = Qwen3MoeSparseMoeBlock(
                layer_id=layer_id,
                config=cfg,
                quant_config=quant_config,
                prefix=add_prefix("mlp", prefix),
            )
        else:
            self.mlp = MellumMLP(
                hidden_size=cfg.hidden_size,
                intermediate_size=cfg.intermediate_size,
                hidden_act=cfg.hidden_act,
                quant_config=quant_config,
                prefix=add_prefix("mlp", prefix),
            )

        is_previous_layer_sparse = _is_sparse(layer_id - 1)

View on GitHub (pinned to 0132848349)

Solutions

Set num_experts to the checkpoint's expert count (e.g. 128) in config.json
Also verify related MoE fields (moe_intermediate_size, num_experts_per_tok) are present and consistent with the checkpoint
If the checkpoint really has no experts, change the layer's mlp_layer_types entry to 'dense'

Example fix

// before
"mlp_layer_types": ["sparse"], "num_experts": 0
// after
"mlp_layer_types": ["sparse"], "num_experts": 128, "moe_intermediate_size": 768

Defensive patterns

Strategy: validation

Validate before calling

cfg = AutoConfig.from_pretrained(path)
if "sparse" in cfg.mlp_layer_types:
    assert getattr(cfg, "num_experts", 0) > 0, "num_experts must be > 0 for sparse layers"

Type guard

def experts_config_ok(cfg) -> bool:
    return "sparse" not in getattr(cfg, "mlp_layer_types", []) or (getattr(cfg, "num_experts", 0) or 0) > 0

Prevention

When switching a layer to sparse, add num_experts and moe_intermediate_size in the same change
Copy MoE fields verbatim from the source checkpoint config

When it happens

Trigger: config.json has any 'sparse' entry in mlp_layer_types but num_experts is 0, negative, or absent; a dense-only Mellum config reused with an mlp_layer_types list containing 'sparse'.

Common situations: Config merging where MoE fields were dropped; converting from a dense baseline model to sparse without adding expert hyperparameters; misaligned config revisions.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/33ed7c8d305ecfeb. Report an issue: GitHub.