{"record":{"id":"33ed7c8d305ecfeb","repo":"sgl-project/sglang","slug":"sparse-mlp-requested-but-num-experts-0-in-mellu","errorCode":null,"errorMessage":"Sparse MLP requested but num_experts <= 0 in Mellum config","messagePattern":"Sparse MLP requested but num_experts <= 0 in Mellum config","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/models/mellum.py","lineNumber":425,"sourceCode":"                f\"{len(mlp_layer_types)} and {cfg.num_hidden_layers}\"\n            )\n\n        def _is_sparse(lid: int) -> bool:\n            if lid < 0 or lid >= cfg.num_hidden_layers:\n                return False\n            mlp_type = mlp_layer_types[lid]\n            if mlp_type not in (\"sparse\", \"dense\"):\n                raise ValueError(\n                    f\"Unsupported mlp_layer_types[{lid}]={mlp_type}; \"\n                    \"expected 'sparse' or 'dense'\"\n                )\n            return mlp_type == \"sparse\"\n\n        self.is_layer_sparse = _is_sparse(layer_id)\n\n        if self.is_layer_sparse:\n            if num_experts <= 0:\n                raise ValueError(\n                    \"Sparse MLP requested but num_experts <= 0 in Mellum config\"\n                )\n            self.mlp = Qwen3MoeSparseMoeBlock(\n                layer_id=layer_id,\n                config=cfg,\n                quant_config=quant_config,\n                prefix=add_prefix(\"mlp\", prefix),\n            )\n        else:\n            self.mlp = MellumMLP(\n                hidden_size=cfg.hidden_size,\n                intermediate_size=cfg.intermediate_size,\n                hidden_act=cfg.hidden_act,\n                quant_config=quant_config,\n                prefix=add_prefix(\"mlp\", prefix),\n            )\n\n        is_previous_layer_sparse = _is_sparse(layer_id - 1)","sourceCodeStart":407,"sourceCodeEnd":443,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/mellum.py#L407-L443","documentation":"When a Mellum layer is marked 'sparse' in mlp_layer_types, the model instantiates a Qwen3MoeSparseMoeBlock which requires config.num_experts > 0. If num_experts is missing (0 or negative), the sparse routing head cannot be built and init aborts.","triggerScenarios":"config.json has any 'sparse' entry in mlp_layer_types but num_experts is 0, negative, or absent; a dense-only Mellum config reused with an mlp_layer_types list containing 'sparse'.","commonSituations":"Config merging where MoE fields were dropped; converting from a dense baseline model to sparse without adding expert hyperparameters; misaligned config revisions.","solutions":["Set num_experts to the checkpoint's expert count (e.g. 128) in config.json","Also verify related MoE fields (moe_intermediate_size, num_experts_per_tok) are present and consistent with the checkpoint","If the checkpoint really has no experts, change the layer's mlp_layer_types entry to 'dense'"],"exampleFix":"// before\n\"mlp_layer_types\": [\"sparse\"], \"num_experts\": 0\n// after\n\"mlp_layer_types\": [\"sparse\"], \"num_experts\": 128, \"moe_intermediate_size\": 768","handlingStrategy":"validation","validationCode":"cfg = AutoConfig.from_pretrained(path)\nif \"sparse\" in cfg.mlp_layer_types:\n    assert getattr(cfg, \"num_experts\", 0) > 0, \"num_experts must be > 0 for sparse layers\"","typeGuard":"def experts_config_ok(cfg) -> bool:\n    return \"sparse\" not in getattr(cfg, \"mlp_layer_types\", []) or (getattr(cfg, \"num_experts\", 0) or 0) > 0","tryCatchPattern":null,"preventionTips":["When switching a layer to sparse, add num_experts and moe_intermediate_size in the same change","Copy MoE fields verbatim from the source checkpoint config"],"tags":["mellum","moe","config-validation","experts"],"backgroundTag":"missing-config-field","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}