vllm-project/vllm · error · ValueError

The Inkling checkpoint does not contain MTP weights

Error message

The Inkling checkpoint does not contain MTP weights

What it means

When enabling MTP speculation on an Inkling checkpoint (model_type inkling_mm_model/inkling_model), vLLM reads mtp_config.num_nextn_predict_layers from the HF config; a value < 1 means the checkpoint ships no MTP head weights, so an MTP draft model cannot be built from it.

Solutions

1. Switch to the Inkling checkpoint variant that includes MTP weights (one whose mtp_config.num_nextn_predict_layers >= 1)
2. Or disable MTP speculation (drop --speculative-method mtp / num_speculative_tokens) for this base model
3. If you merged MTP weights yourself, ensure mtp_config in config.json declares the layer count

Example fix

# before
vllm serve inkling/inkling-base --speculative-method mtp --num-speculative-tokens 1

# after
vllm serve inkling/inkling-mtp --num-speculative-tokens 1

from transformers import AutoConfig

def inkling_has_mtp(model_id: str) -> bool:
    cfg = AutoConfig.from_pretrained(model_id)
    return (getattr(cfg, 'mtp_config', None) or {}).get('num_nextn_predict_layers', 0) >= 1

null

null

When it happens

Trigger: Running --speculative-method mtp against a base Inkling checkpoint released without the MTP layers; mtp_config missing or empty {} in config.json (defaults to 0); a distilled/pruned variant that drops MTP weights.

Common situations: Assuming every release of a family includes MTP layers; using the chat model instead of the reasoning/MTP variant; config.json fields renamed between releases.

Related errors

• mtp_layer_types must have one entry per MTP layer: got
• target_model_config must be present for mtp
• A speculative model was provided, but…
• dspark_draft_topk is only supported by DSpark
• Either prompt_lookup_max or prompt_lookup_min must be…

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/7ad46c07073a2dc3. Report an issue: GitHub.

Appendix: source

Thrown at vllm/config/speculative.py:629

            n_predict = getattr(hf_config, "num_nextn_predict_layers", 1)
            hf_config.update({"n_predict": n_predict, "architectures": ["Step3p5MTP"]})

        if initial_architecture == "MistralLarge3ForCausalLM":
            hf_config.update({"architectures": ["EagleMistralLarge3ForCausalLM"]})

        if hf_config.model_type == "hy_v3":
            hf_config.model_type = "hy_v3_mtp"
            n_predict = getattr(hf_config, "num_nextn_predict_layers", None)
            hf_config.update(
                {"n_predict": n_predict, "architectures": ["HYV3MTPModel"]}
            )

        if hf_config.model_type in ("inkling_mm_model", "inkling_model"):
            mtp_config = getattr(hf_config, "mtp_config", None) or {}
            hf_config = getattr(hf_config, "text_config", hf_config)
            checkpoint_depths = mtp_config.get("num_nextn_predict_layers", 0)
            if checkpoint_depths < 1:
                raise ValueError("The Inkling checkpoint does not contain MTP weights")
            hf_config.model_type = "inkling_mtp"
            hf_config.update(
                {
                    "n_predict": checkpoint_depths,
                    "num_nextn_predict_layers": checkpoint_depths,
                    "chain_hidden_post_norm": mtp_config.get(
                        "chain_hidden_post_norm", False
                    ),
                    "local_layer_ids": mtp_config.get("local_layer_ids", []),
                    "architectures": ["InklingMTPModel"],
                }
            )

        if hf_config.model_type in ("gemma4_assistant", "gemma4_unified_assistant"):
            hf_config.model_type = "gemma4_mtp"
            text_config = getattr(hf_config, "text_config", hf_config)
            # The assistant runs all decoder layers in a single forward
            # call to produce one draft token, so n_predict=1.

View on GitHub (pinned to c794754062)