{"record":{"id":"f644a7c8e4fdc46c","repo":"vllm-project/vllm","slug":"total-number-of-attention-heads-total-num-attent","errorCode":null,"errorMessage":"Total number of attention heads ({total_num_attention_heads}) must be divisible by tensor parallel size ({tensor_parallel_size}).","messagePattern":"Total number of attention heads \\((.+?)\\) must be divisible by tensor parallel size \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/model.py","lineNumber":1368,"sourceCode":"                self.hf_config.dual_chunk_attention_config[\n                    \"sparse_attention_config\"\n                ] = sparse_attn_config\n                if (\n                    \"sparse_attention_enabled\"\n                    not in self.hf_config.dual_chunk_attention_config\n                ):\n                    self.hf_config.dual_chunk_attention_config[\n                        \"sparse_attention_enabled\"\n                    ] = True\n\n    def verify_with_parallel_config(\n        self,\n        parallel_config: ParallelConfig,\n    ) -> None:\n        total_num_attention_heads = self.model_arch_config.total_num_attention_heads\n        tensor_parallel_size = parallel_config.tensor_parallel_size\n        if total_num_attention_heads % tensor_parallel_size != 0:\n            raise ValueError(\n                f\"Total number of attention heads ({total_num_attention_heads})\"\n                \" must be divisible by tensor parallel size \"\n                f\"({tensor_parallel_size}).\"\n            )\n\n        if parallel_config.enable_expert_parallel:\n            self._verify_with_expert_parallelism()\n\n        pipeline_parallel_size = parallel_config.pipeline_parallel_size\n        if pipeline_parallel_size > 1 and not self.registry.is_pp_supported_model(\n            self.architectures, self\n        ):\n            raise NotImplementedError(\n                \"Pipeline parallelism is not supported for this model. \"\n                \"Supported models implement the `SupportsPP` interface.\"\n            )\n\n        decode_context_parallel_size = parallel_config.decode_context_parallel_size","sourceCodeStart":1350,"sourceCodeEnd":1386,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/model.py#L1350-L1386","documentation":"Raised by verify_with_parallel_config when the model's total attention heads are not evenly divisible by the tensor parallel size. TP shards attention heads across GPUs, so each rank must receive an integer number of heads.","triggerScenarios":"Calling ModelConfig.verify_with_parallel_config with tensor_parallel_size=N where model_arch_config.total_num_attention_heads % N != 0 (e.g. 40-head model with TP=3).","commonSituations":"Scaling TP beyond what the head count allows (e.g. TP=8 on a 32-head model is fine, TP=6 is not); using a custom fine-tuned checkpoint with an unusual head count; mixing up head count with hidden size when computing max TP.","solutions":["Set --tensor-parallel-size to a divisor of the attention head count (e.g. for 40 heads: 1, 2, 4, 5, 8, 10, 20, 40).","Check the model's num_attention_heads in config.json and divide down from there before choosing TP.","If more GPUs are needed than TP allows, combine TP with pipeline parallelism or data parallelism instead."],"exampleFix":"# before (40 heads)\nvllm serve Qwen/Qwen2.5-32B --tensor-parallel-size 3\n# after\nvllm serve Qwen/Qwen2.5-32B --tensor-parallel-size 4","handlingStrategy":"validation","validationCode":"def valid_tp_sizes(num_heads: int):\n    return [tp for tp in range(1, num_heads + 1) if num_heads % tp == 0]\n# pick tensor_parallel_size from valid_tp_sizes(config.num_attention_heads)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Read num_attention_heads from config.json and choose TP among its divisors before launch.","Automate launch-flag generation from the model config instead of hardcoding TP.","Script a preflight check: assert heads % tp == 0 in deployment templates."],"tags":["tensor-parallelism","attention","config","startup"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}