vllm-project/vllm · error · ValueError

{speculative_draft_tensor_parallel_size=} cannot be other va

Error message

{speculative_draft_tensor_parallel_size=} cannot be other value than 1 or target model tensor_parallel_size

What it means

Raised by SpeculativeConfig._verify_and_get_draft_tp when the resolved speculative_draft_tensor_parallel_size is neither 1 nor the target's tensor_parallel_size. The draft engine must run either fully replicated (TP=1) or with the same sharding as the target, because the draft and target weights/activations are exchanged each step; any other factor has no supported topology.

Source

Thrown at vllm/config/speculative.py:1290

        # appropriately else verify that it is set correctly.
        if speculative_draft_tensor_parallel_size is None:
            if draft_hf_config.model_type == "mlp_speculator":
                speculative_draft_tensor_parallel_size = 1
                if target_parallel_config.tensor_parallel_size > 1:
                    logger.warning(
                        "%s cannot currently be run with tp>1; "
                        "setting speculative_draft_tensor_parallel_size=1",
                        draft_hf_config.model_type,
                    )
            else:
                speculative_draft_tensor_parallel_size = (
                    target_parallel_config.tensor_parallel_size
                )
        elif speculative_draft_tensor_parallel_size not in (
            1,
            target_parallel_config.tensor_parallel_size,
        ):
            raise ValueError(
                f"{speculative_draft_tensor_parallel_size=} cannot be "
                f"other value than 1 or target model tensor_parallel_size"
            )
        return speculative_draft_tensor_parallel_size

    def update_arch_(self):
        """
        EagleConfig and ExtractHiddenStatesConfig update architectures, so update all
        architectures-related fields in self.draft_model_config
        """
        self.draft_model_config.hf_text_config = get_hf_text_config(
            self.draft_model_config.hf_config
        )
        self.draft_model_config.model_arch_config = (
            self.draft_model_config.get_model_arch_config()
        )
        model_info, arch = self.draft_model_config.registry.inspect_model_cls(
            self.draft_model_config.architectures,

View on GitHub (pinned to c794754062)

Solutions

1. Set speculative_draft_tensor_parallel_size to 1 or to the target's tensor_parallel_size
2. Remove the key entirely to let vLLM auto-resolve it (1 for models that cannot run TP>1, else target TP)

Example fix

# before
speculative_config={"method": "eagle", "model": "...", "speculative_draft_tensor_parallel_size": 2}  # target TP=4
# after
speculative_config={"method": "eagle", "model": "...", "speculative_draft_tensor_parallel_size": 4}

tp = engine_args.tensor_parallel_size
if (dtp := spec_cfg.get("speculative_draft_tensor_parallel_size")) not in (None, 1, tp):
    raise ValueError(f"draft TP must be 1 or {tp}, got {dtp}")

def is_valid_draft_tp(draft_tp: int | None, target_tp: int) -> bool:
    return draft_tp is None or draft_tp in (1, target_tp)

When it happens

Trigger: Passing speculative_draft_tensor_parallel_size=2 with target tensor_parallel_size=4 (or 8 vs 1, etc.) in the speculative_config, after the auto-default branch has not replaced it.

Common situations: Assuming the draft can be sharded independently of the target; leftover values from configs written for a different TP layout when scaling a deployment up or down.

Related errors

• 'tensor_parallel_size' is not a valid argument in the specul
• Total number of attention heads ({total_num_attention_heads}
• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}
• synthetic_acceptance_rates entries must be in [0, 1], got {r

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/c3dd396657c2440f. Report an issue: GitHub.