vllm-project/vllm · error · ValueError

Total number of attention heads ({total_num_attention_heads}

Error message

Total number of attention heads ({total_num_attention_heads}) must be divisible by tensor parallel size ({tensor_parallel_size}).

What it means

Raised by verify_with_parallel_config when the model's total attention heads are not evenly divisible by the tensor parallel size. TP shards attention heads across GPUs, so each rank must receive an integer number of heads.

Source

Thrown at vllm/config/model.py:1368

                self.hf_config.dual_chunk_attention_config[
                    "sparse_attention_config"
                ] = sparse_attn_config
                if (
                    "sparse_attention_enabled"
                    not in self.hf_config.dual_chunk_attention_config
                ):
                    self.hf_config.dual_chunk_attention_config[
                        "sparse_attention_enabled"
                    ] = True

    def verify_with_parallel_config(
        self,
        parallel_config: ParallelConfig,
    ) -> None:
        total_num_attention_heads = self.model_arch_config.total_num_attention_heads
        tensor_parallel_size = parallel_config.tensor_parallel_size
        if total_num_attention_heads % tensor_parallel_size != 0:
            raise ValueError(
                f"Total number of attention heads ({total_num_attention_heads})"
                " must be divisible by tensor parallel size "
                f"({tensor_parallel_size})."
            )

        if parallel_config.enable_expert_parallel:
            self._verify_with_expert_parallelism()

        pipeline_parallel_size = parallel_config.pipeline_parallel_size
        if pipeline_parallel_size > 1 and not self.registry.is_pp_supported_model(
            self.architectures, self
        ):
            raise NotImplementedError(
                "Pipeline parallelism is not supported for this model. "
                "Supported models implement the `SupportsPP` interface."
            )

        decode_context_parallel_size = parallel_config.decode_context_parallel_size

View on GitHub (pinned to c794754062)

Solutions

1. Set --tensor-parallel-size to a divisor of the attention head count (e.g. for 40 heads: 1, 2, 4, 5, 8, 10, 20, 40).
2. Check the model's num_attention_heads in config.json and divide down from there before choosing TP.
3. If more GPUs are needed than TP allows, combine TP with pipeline parallelism or data parallelism instead.

Example fix

# before (40 heads)
vllm serve Qwen/Qwen2.5-32B --tensor-parallel-size 3
# after
vllm serve Qwen/Qwen2.5-32B --tensor-parallel-size 4

def valid_tp_sizes(num_heads: int):
    return [tp for tp in range(1, num_heads + 1) if num_heads % tp == 0]
# pick tensor_parallel_size from valid_tp_sizes(config.num_attention_heads)

When it happens

Trigger: Calling ModelConfig.verify_with_parallel_config with tensor_parallel_size=N where model_arch_config.total_num_attention_heads % N != 0 (e.g. 40-head model with TP=3).

Common situations: Scaling TP beyond what the head count allows (e.g. TP=8 on a 32-head model is fine, TP=6 is not); using a custom fine-tuned checkpoint with an unusual head count; mixing up head count with hidden size when computing max TP.

Related errors

• cannot use in-process coordinator with bootstrapped transpor
• The quantization method %s is deprecated and will be removed
• Number of experts in the model must be greater than 0 when e
• Pipeline parallelism is not supported for this model. Suppor
• The model is an hybrid without a layers_block_type or an att

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f644a7c8e4fdc46c. Report an issue: GitHub.