{"record":{"id":"2c595fa3e19176e3","repo":"huggingface/transformers","slug":"number-of-key-value-heads-must-be-divisible-by","errorCode":null,"errorMessage":"Number of key value heads {} must be divisible by tensor parallel size {}.","messagePattern":"Number of key value heads (.+?) must be divisible by tensor parallel size (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":207,"sourceCode":"            sliding_window = config.sliding_window if group_types[i] == \"sliding_attention\" else 1\n            for j, layer in enumerate(group):\n                self.layer_index_to_group_indices[layer] = (i, j)\n                self.sliding_windows[layer] = sliding_window\n\n        # Check if the KV heads are part of the TP plan. If they are not, the cache does not need plan for TP.\n        # TODO: this is fragile. If your model fails to TP properly because of this, please open an issue.\n        kv_is_tp = True\n        for key in [\"layers.*.self_attn.k_proj\", \"layers.*.self_attn.v_proj\"]:\n            if not (key in tp_plan or \"model.\" + key in tp_plan):\n                kv_is_tp = False\n                break\n\n        # If the KV heads are TP'ed, each KV head is dispatched to a different GPU, so the effective number of KV heads\n        # per GPU is simply divided by the TP size\n        tp_size = distributed_helper.tp_size\n        if tp_size > 1 and kv_is_tp:\n            if self.num_key_value_heads % tp_size != 0:\n                raise ValueError(\n                    f\"Number of key value heads {self.num_key_value_heads} must be divisible by tensor parallel size {tp_size}.\"\n                )\n            self.num_key_value_heads //= tp_size\n\n        # If somehow the max memory percent is not yet resolved, resolve it conservatively\n        if continuous_batching_config.max_memory_percent is None:\n            resolve_max_memory_percent(cb_config=continuous_batching_config, has_logit_processors=True)\n\n        max_batch_tokens, num_blocks = PagedAttentionMemoryHandler(\n            config=config,\n            continuous_batching_config=continuous_batching_config,\n            dtype=self.dtype,\n            group_types=group_types,\n            group_size=group_size,\n        ).infer_max_batch_tokens_and_num_blocks()\n\n        # For TP, align max_batch_tokens and num_blocks to the minimal value across the TP group\n        if tp_size > 1:","sourceCodeStart":189,"sourceCodeEnd":225,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L189-L225","documentation":"ValueError raised when tensor parallelism shards the K/V projections (tp_plan contains layers.*.self_attn.k_proj / v_proj) but config.num_key_value_heads is not divisible by the TP size. Each GPU must receive a whole number of KV heads for GQA sharding; otherwise attention shapes become inconsistent across ranks.","triggerScenarios":"Running multi-GPU continuous batching with tensor_parallel_size=N where N does not divide num_key_value_heads (e.g. 7 KV heads with TP=4); models with prime/small KV-head counts (e.g. num_key_value_heads=8 with TP=6); custom tp_plan strings that accidentally match k_proj/v_proj.","commonSituations":"Scaling a GQA model (Llama-style with few KV heads) across more GPUs than the KV head count divides; mixing tp_plan styles ('model.layers...' vs 'layers...') so KV sharding is detected unexpectedly.","solutions":["Choose a tensor_parallel_size that divides num_key_value_heads (e.g. for 8 KV heads use TP in {1,2,4,8})","Or use data/pipeline parallelism instead of TP for models with few KV heads","Verify config.num_key_value_heads and the tp_plan; if KV projections should be replicated, remove them from the TP plan"],"exampleFix":"# before\ndevices = ['cuda:0','cuda:1','cuda:2','cuda:3']  # TP=4, model has num_key_value_heads=8 -> ok; with 6 heads -> error\n# after: pick TP that divides KV heads\ndevices = ['cuda:0','cuda:1']  # TP=2 divides 6 and 8","handlingStrategy":"validation","validationCode":"kv = getattr(config, 'num_key_value_heads', None) or config.num_attention_heads\nfor tp in candidate_tp_sizes:\n    if kv % tp == 0:\n        return tp\nraise ValueError(f'no valid TP size divides {kv} KV heads')","typeGuard":"def tp_compatible(config, tp_size: int) -> bool:\n    kv = getattr(config, 'num_key_value_heads', None) or config.num_attention_heads\n    return tp_size <= 1 or kv % tp_size == 0","tryCatchPattern":null,"preventionTips":["Pick TP sizes from the divisors of num_key_value_heads","Models with few KV heads (GQA) often cannot use high TP — prefer DP","Double-check tp_plan entries for k_proj/v_proj when you intend replication"],"tags":["continuous-batching","tensor-parallelism","multi-gpu","gqa","kv-cache"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}