huggingface/transformers · error · ValueError

num_key_value_heads or num_attention_heads could not be foun

Error message

num_key_value_heads or num_attention_heads could not be found in the config:
{}

What it means

ValueError from find_num_kv_heads() in the continuous-batching (vLLM-style) cache: it tries config.num_key_value_heads first (GQA models), then falls back to config.num_attention_heads; if both attributes are missing from the model config, the KV cache cannot size its tensors and the error is raised with the config dump appended.

Source

Thrown at src/transformers/generation/continuous_batching/cache.py:39

from ...generation.configuration_utils import ContinuousBatchingConfig
from ...utils.generic import is_flash_attention_requested
from .cache_manager import BlockManager, CacheAllocator, FullAttentionCacheAllocator, SlidingAttentionCacheAllocator
from .distributed import DistributedHelper
from .initialization import resolve_max_memory_percent
from .requests import RequestState, RequestStatus, get_device_and_memory_breakdown, logger


def find_num_kv_heads(config: PreTrainedConfig) -> int:
    """Finds the number of key-value heads for the given config."""
    # If the model supports GQA, we leverage it by using the num_key_value_heads attribute
    kv_heads = getattr(config, "num_key_value_heads", None)
    if kv_heads is not None:
        return kv_heads
    # Otherwise, the number of KV heads is the same as the number of attention heads
    kv_heads = getattr(config, "num_attention_heads", None)
    if kv_heads is not None:
        return kv_heads
    raise ValueError(f"num_key_value_heads or num_attention_heads could not be found in the config:\n{config}")


def find_head_dim(config: PreTrainedConfig) -> int:
    """Finds the head dimension for the given config."""
    # If the model has the head_dim attribute, there is nothing to do but return it
    head_dim = getattr(config, "head_dim", None)
    if head_dim is not None:
        return head_dim
    # If it is missing, we may reconstruct it from the hidden size and the number of attention heads
    hidden_size = getattr(config, "hidden_size", None)
    num_attention_heads = getattr(config, "num_attention_heads", None)
    if hidden_size is not None and num_attention_heads is not None:
        return hidden_size // num_attention_heads
    raise ValueError(f"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\n{config}")


def group_layers_by_attn_type(config: PreTrainedConfig) -> tuple[list[list[int]], list[str]]:
    """

View on GitHub (pinned to a597f97485)

Solutions

1. Ensure the config passed to the cache exposes num_attention_heads (standard text models) or num_key_value_heads (GQA)
2. For multi-modal models, pass the language-model sub-config (e.g. config.text_config) rather than the top-level config
3. Set the attribute explicitly before init: config.num_attention_heads = config.n_heads if your config uses a different name

Example fix

# before
cache = Cache(config)  # top-level multimodal config, missing num_attention_heads
# after
cache = Cache(config.text_config)  # or: config.num_attention_heads = 32 before init

assert getattr(config, 'num_key_value_heads', None) or getattr(config, 'num_attention_heads', None), \
    'config must define num_attention_heads or num_key_value_heads'

def has_kv_head_info(config) -> bool:
    return getattr(config, 'num_key_value_heads', None) is not None or getattr(config, 'num_attention_heads', None) is not None

When it happens

Trigger: Passing a custom or exotic PreTrainedConfig lacking num_attention_heads (e.g. some vision/multi-modal backbones or hand-built configs) into the continuous-batching generate path; a config class that names the field differently (n_heads, num_heads).

Common situations: Wiring new architectures into continuous batching; loading models whose attention config lives on sub-configs (text_config) and the outer config is passed instead.

Related errors

• head_dim or (hidden_size and num_attention_heads) could not
• Number of key value heads {} must be divisible by tensor par
• Invalid group type: {}
• Block size must be at least {}, but got {}
• Failed to allocate {} blocks for request {}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2f8a8fe9d2b7f4bc. Report an issue: GitHub.