{"record":{"id":"c3dd396657c2440f","repo":"vllm-project/vllm","slug":"speculative-draft-tensor-parallel-size-cannot-b","errorCode":null,"errorMessage":"{speculative_draft_tensor_parallel_size=} cannot be other value than 1 or target model tensor_parallel_size","messagePattern":"(.+?) cannot be other value than 1 or target model tensor_parallel_size","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/speculative.py","lineNumber":1290,"sourceCode":"        # appropriately else verify that it is set correctly.\n        if speculative_draft_tensor_parallel_size is None:\n            if draft_hf_config.model_type == \"mlp_speculator\":\n                speculative_draft_tensor_parallel_size = 1\n                if target_parallel_config.tensor_parallel_size > 1:\n                    logger.warning(\n                        \"%s cannot currently be run with tp>1; \"\n                        \"setting speculative_draft_tensor_parallel_size=1\",\n                        draft_hf_config.model_type,\n                    )\n            else:\n                speculative_draft_tensor_parallel_size = (\n                    target_parallel_config.tensor_parallel_size\n                )\n        elif speculative_draft_tensor_parallel_size not in (\n            1,\n            target_parallel_config.tensor_parallel_size,\n        ):\n            raise ValueError(\n                f\"{speculative_draft_tensor_parallel_size=} cannot be \"\n                f\"other value than 1 or target model tensor_parallel_size\"\n            )\n        return speculative_draft_tensor_parallel_size\n\n    def update_arch_(self):\n        \"\"\"\n        EagleConfig and ExtractHiddenStatesConfig update architectures, so update all\n        architectures-related fields in self.draft_model_config\n        \"\"\"\n        self.draft_model_config.hf_text_config = get_hf_text_config(\n            self.draft_model_config.hf_config\n        )\n        self.draft_model_config.model_arch_config = (\n            self.draft_model_config.get_model_arch_config()\n        )\n        model_info, arch = self.draft_model_config.registry.inspect_model_cls(\n            self.draft_model_config.architectures,","sourceCodeStart":1272,"sourceCodeEnd":1308,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/speculative.py#L1272-L1308","documentation":"Raised by SpeculativeConfig._verify_and_get_draft_tp when the resolved speculative_draft_tensor_parallel_size is neither 1 nor the target's tensor_parallel_size. The draft engine must run either fully replicated (TP=1) or with the same sharding as the target, because the draft and target weights/activations are exchanged each step; any other factor has no supported topology.","triggerScenarios":"Passing speculative_draft_tensor_parallel_size=2 with target tensor_parallel_size=4 (or 8 vs 1, etc.) in the speculative_config, after the auto-default branch has not replaced it.","commonSituations":"Assuming the draft can be sharded independently of the target; leftover values from configs written for a different TP layout when scaling a deployment up or down.","solutions":["Set speculative_draft_tensor_parallel_size to 1 or to the target's tensor_parallel_size","Remove the key entirely to let vLLM auto-resolve it (1 for models that cannot run TP>1, else target TP)"],"exampleFix":"# before\nspeculative_config={\"method\": \"eagle\", \"model\": \"...\", \"speculative_draft_tensor_parallel_size\": 2}  # target TP=4\n# after\nspeculative_config={\"method\": \"eagle\", \"model\": \"...\", \"speculative_draft_tensor_parallel_size\": 4}","handlingStrategy":"validation","validationCode":"tp = engine_args.tensor_parallel_size\nif (dtp := spec_cfg.get(\"speculative_draft_tensor_parallel_size\")) not in (None, 1, tp):\n    raise ValueError(f\"draft TP must be 1 or {tp}, got {dtp}\")","typeGuard":"def is_valid_draft_tp(draft_tp: int | None, target_tp: int) -> bool:\n    return draft_tp is None or draft_tp in (1, target_tp)","tryCatchPattern":null,"preventionTips":["Omit speculative_draft_tensor_parallel_size and let vLLM auto-resolve it","When scaling target TP up/down, re-validate (or delete) the draft TP override in the saved config"],"tags":["speculative-decoding","tensor-parallelism","draft-model","config"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}