{"record":{"id":"686ffa612aa6a436","repo":"huggingface/transformers","slug":"could-not-find-num-mtp-layers-in-the-model-confi","errorCode":null,"errorMessage":"Could not find `num_mtp_layers` in the model config. This model probably has no associated mtp weights.","messagePattern":"Could not find `num_mtp_layers` in the model config\\. This model probably has no associated mtp weights\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/candidate_generator.py","lineNumber":1435,"sourceCode":"\nclass MTPCandidateGenerator(AssistedCandidateGenerator):\n    requires_model_outputs: bool = True\n    # We always need to pass the hidden states from the main model\n    model_kwargs_overrides: dict[str, Any] = {\"output_hidden_states\": True}\n\n    def __init__(\n        self,\n        main_model: \"PreTrainedModel\",\n        generation_config: \"GenerationConfig\",\n        model_kwargs: dict[str, Any],\n        logits_processor: Optional[\"LogitsProcessorList\"] = None,\n    ):\n        from ..cache_utils import MtpCache\n        from ..modeling_layers import MtpModel\n\n        self.num_mtp_layers = getattr(main_model.config.get_text_config(), \"num_mtp_layers\", None)\n        if self.num_mtp_layers is None:\n            raise ValueError(\n                \"Could not find `num_mtp_layers` in the model config. This model probably has no associated \"\n                \"mtp weights.\"\n            )\n\n        # Heuristic: use the device of the last layer of the main model for the MTP layers\n        base_model = main_model.get_decoder()\n        self.device = next(x.device for x in base_model.layers[-1].parameters())  # type: ignore\n        self.mtp_model = MtpModel.from_pretrained(main_model, device_map={\"\": self.device})\n\n        # Create the mtp cache and allow it to keep its past before we crop it\n        self.mtp_cache = MtpCache(config=main_model.config.get_mtp_config())\n        self.mtp_cache.activate_past_recording()\n\n        # Save those to know how to decode mtp tokens\n        self.do_sample = generation_config.do_sample\n        self.logits_processor = logits_processor\n\n        self.is_main_model_prefill = True","sourceCodeStart":1417,"sourceCodeEnd":1453,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/candidate_generator.py#L1417-L1453","documentation":"MtpCandidateGenerator implements multi-token-prediction (MTP) speculative decoding: it builds extra MTP layers from the checkpoint via MtpModel.from_pretrained. It keys off config.get_text_config().num_mtp_layers; if that attribute is absent, the checkpoint has no MTP head weights and the generator cannot work.","triggerScenarios":"Selecting the MTP candidate generator for a model whose config lacks num_mtp_layers — e.g. pointing DeepSeek-V3-style MTP decoding at a regular checkpoint, or a community upload that stripped MTP weights/config fields.","commonSituations":"Enabling MTP speculative decoding on a model that was not trained/exported with MTP layers, or a config.json missing the num_mtp_layers key after conversion/quantization.","solutions":["Use a checkpoint that actually ships MTP layers (its config.json contains num_mtp_layers > 0 and the checkpoint has the MTP weights)","Fall back to standard assisted decoding with a separate small draft model if MTP weights are unavailable","Do not manually select the MTP generator; let the generation config/model decide when MTP is present"],"exampleFix":"# before: model without MTP layers, MTP generator forced -> ValueError\n# after: verify before wiring\nnum_mtp = getattr(model.config.get_text_config(), \"num_mtp_layers\", None)\nif num_mtp:\n    generator = MtpCandidateGenerator(main_model=model, generation_config=cfg, model_kwargs=kw)\nelse:\n    generator = None  # use standard assisted decoding","handlingStrategy":"validation","validationCode":"def model_has_mtp(main_model) -> bool:\n    return getattr(main_model.config.get_text_config(), \"num_mtp_layers\", None) is not None","typeGuard":"def supports_mtp_generation(model) -> bool:\n    cfg = getattr(model.config, \"get_text_config\", lambda: model.config)()\n    return isinstance(getattr(cfg, \"num_mtp_layers\", None), int) and cfg.num_mtp_layers > 0","tryCatchPattern":null,"preventionTips":["Check config.json for num_mtp_layers before enabling MTP speculative decoding","Keep a fallback code path (standard assisted decoding) for checkpoints without MTP weights"],"tags":["python","transformers","generation","mtp","speculative-decoding","model-config"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}