{"record":{"id":"731582cf70d243ef","repo":"sgl-project/sglang","slug":"expected-len-mlp-layer-types-num-hidden-layers","errorCode":null,"errorMessage":"Expected len(mlp_layer_types) == num_hidden_layers, got {len(mlp_layer_types)} and {cfg.num_hidden_layers}","messagePattern":"Expected len\\(mlp_layer_types\\) == num_hidden_layers, got (.+?) and (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/models/mellum.py","lineNumber":405,"sourceCode":"            max_position_embeddings=max_position_embeddings,\n            head_dim=head_dim,\n            rms_norm_eps=rms_norm_eps,\n            attention_bias=attention_bias,\n            config=config,\n            quant_config=quant_config,\n            prefix=add_prefix(\"self_attn\", prefix),\n            sliding_window_size=sliding_window_size,\n            alt_stream=alt_stream,\n        )\n\n        self.attn_tp_size = get_parallel().attn_tp_size\n        self.attn_tp_rank = get_parallel().attn_tp_rank\n\n        mlp_layer_types = cfg.mlp_layer_types\n        num_experts = cfg.num_experts\n\n        if len(mlp_layer_types) != cfg.num_hidden_layers:\n            raise ValueError(\n                \"Expected len(mlp_layer_types) == num_hidden_layers, got \"\n                f\"{len(mlp_layer_types)} and {cfg.num_hidden_layers}\"\n            )\n\n        def _is_sparse(lid: int) -> bool:\n            if lid < 0 or lid >= cfg.num_hidden_layers:\n                return False\n            mlp_type = mlp_layer_types[lid]\n            if mlp_type not in (\"sparse\", \"dense\"):\n                raise ValueError(\n                    f\"Unsupported mlp_layer_types[{lid}]={mlp_type}; \"\n                    \"expected 'sparse' or 'dense'\"\n                )\n            return mlp_type == \"sparse\"\n\n        self.is_layer_sparse = _is_sparse(layer_id)\n\n        if self.is_layer_sparse:","sourceCodeStart":387,"sourceCodeEnd":423,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/mellum.py#L387-L423","documentation":"Mellum supports per-layer sparse (MoE) vs dense MLPs driven by config.mlp_layer_types, which must list one entry per hidden layer. The model validates len(mlp_layer_types) == num_hidden_layers at init and rejects mismatches. A mismatch means the config is internally inconsistent about layer count.","triggerScenarios":"config.json has mlp_layer_types of a different length than num_hidden_layers, e.g. after pruning layers, editing num_hidden_layers, or pasting an mlp_layer_types list from another model size.","commonSituations":"Fine-tuning scripts that override num_hidden_layers; manual config surgery when swapping MoE layers; mismatched config/checkpoint revisions of Mellum.","solutions":["Regenerate mlp_layer_types so it has exactly num_hidden_layers entries (e.g. [\"dense\",\"sparse\",...])","Or correct num_hidden_layers to match the checkpoint's actual layer count","Prefer loading the original config.json shipped with the checkpoint rather than merging configs"],"exampleFix":"// before\n\"num_hidden_layers\": 28, \"mlp_layer_types\": [\"sparse\"] * 27\n// after\n\"num_hidden_layers\": 28, \"mlp_layer_types\": [\"dense\"] + [\"sparse\"] * 27","handlingStrategy":"validation","validationCode":"cfg = AutoConfig.from_pretrained(path)\nassert len(cfg.mlp_layer_types) == cfg.num_hidden_layers, (\n    len(cfg.mlp_layer_types), cfg.num_hidden_layers)","typeGuard":"def mlp_types_consistent(cfg) -> bool:\n    ml = getattr(cfg, \"mlp_layer_types\", None)\n    return ml is not None and len(ml) == cfg.num_hidden_layers","tryCatchPattern":null,"preventionTips":["Regenerate mlp_layer_types programmatically whenever num_hidden_layers changes","Keep config fields from one source; avoid merging configs across model sizes"],"tags":["mellum","config-validation","moe","layer-config"],"backgroundTag":"config-length-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}