huggingface/transformers · error · ValueError

Could not find `num_mtp_layers` in the model config. This mo

Error message

Could not find `num_mtp_layers` in the model config. This model probably has no associated mtp weights.

What it means

MtpCandidateGenerator implements multi-token-prediction (MTP) speculative decoding: it builds extra MTP layers from the checkpoint via MtpModel.from_pretrained. It keys off config.get_text_config().num_mtp_layers; if that attribute is absent, the checkpoint has no MTP head weights and the generator cannot work.

Source

Thrown at src/transformers/generation/candidate_generator.py:1435

class MTPCandidateGenerator(AssistedCandidateGenerator):
    requires_model_outputs: bool = True
    # We always need to pass the hidden states from the main model
    model_kwargs_overrides: dict[str, Any] = {"output_hidden_states": True}

    def __init__(
        self,
        main_model: "PreTrainedModel",
        generation_config: "GenerationConfig",
        model_kwargs: dict[str, Any],
        logits_processor: Optional["LogitsProcessorList"] = None,
    ):
        from ..cache_utils import MtpCache
        from ..modeling_layers import MtpModel

        self.num_mtp_layers = getattr(main_model.config.get_text_config(), "num_mtp_layers", None)
        if self.num_mtp_layers is None:
            raise ValueError(
                "Could not find `num_mtp_layers` in the model config. This model probably has no associated "
                "mtp weights."
            )

        # Heuristic: use the device of the last layer of the main model for the MTP layers
        base_model = main_model.get_decoder()
        self.device = next(x.device for x in base_model.layers[-1].parameters())  # type: ignore
        self.mtp_model = MtpModel.from_pretrained(main_model, device_map={"": self.device})

        # Create the mtp cache and allow it to keep its past before we crop it
        self.mtp_cache = MtpCache(config=main_model.config.get_mtp_config())
        self.mtp_cache.activate_past_recording()

        # Save those to know how to decode mtp tokens
        self.do_sample = generation_config.do_sample
        self.logits_processor = logits_processor

        self.is_main_model_prefill = True

View on GitHub (pinned to a597f97485)

Solutions

1. Use a checkpoint that actually ships MTP layers (its config.json contains num_mtp_layers > 0 and the checkpoint has the MTP weights)
2. Fall back to standard assisted decoding with a separate small draft model if MTP weights are unavailable
3. Do not manually select the MTP generator; let the generation config/model decide when MTP is present

Example fix

# before: model without MTP layers, MTP generator forced -> ValueError
# after: verify before wiring
num_mtp = getattr(model.config.get_text_config(), "num_mtp_layers", None)
if num_mtp:
    generator = MtpCandidateGenerator(main_model=model, generation_config=cfg, model_kwargs=kw)
else:
    generator = None  # use standard assisted decoding

def model_has_mtp(main_model) -> bool:
    return getattr(main_model.config.get_text_config(), "num_mtp_layers", None) is not None

def supports_mtp_generation(model) -> bool:
    cfg = getattr(model.config, "get_text_config", lambda: model.config)()
    return isinstance(getattr(cfg, "num_mtp_layers", None), int) and cfg.num_mtp_layers > 0

When it happens

Trigger: Selecting the MTP candidate generator for a model whose config lacks num_mtp_layers — e.g. pointing DeepSeek-V3-style MTP decoding at a regular checkpoint, or a community upload that stripped MTP weights/config fields.

Common situations: Enabling MTP speculative decoding on a model that was not trained/exported with MTP layers, or a config.json missing the num_mtp_layers key after conversion/quantization.

Related errors

• Expected assistant_model to be a Gemma4AssistantForCausalLM
• `model_outputs` cannot be None, and they need to contain `hi
• `model_outputs` cannot be None, and they need to contain `hi
• `assistant_ensemble_weight` must be in the open interval `(0
• Model {cls.__name__} has no config class or model type

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/686ffa612aa6a436. Report an issue: GitHub.