huggingface/transformers · error · ValueError

num_key_value_heads or num_attention_heads could not be…

Error message

num_key_value_heads or num_attention_heads could not be found in the config:
{}

What it means

ValueError from find_num_kv_heads() in the continuous-batching (vLLM-style) cache: it tries config.num_key_value_heads first (GQA models), then falls back to config.num_attention_heads; if both attributes are missing from the model config, the KV cache cannot size its tensors and the error is raised with the config dump appended.

Solutions

1. Ensure the config passed to the cache exposes num_attention_heads (standard text models) or num_key_value_heads (GQA)
2. For multi-modal models, pass the language-model sub-config (e.g. config.text_config) rather than the top-level config
3. Set the attribute explicitly before init: config.num_attention_heads = config.n_heads if your config uses a different name

Example fix

# before
cache = Cache(config)  # top-level multimodal config, missing num_attention_heads
# after
cache = Cache(config.text_config)  # or: config.num_attention_heads = 32 before init

assert getattr(config, 'num_key_value_heads', None) or getattr(config, 'num_attention_heads', None), \
    'config must define num_attention_heads or num_key_value_heads'

def has_kv_head_info(config) -> bool:
    return getattr(config, 'num_key_value_heads', None) is not None or getattr(config, 'num_attention_heads', None) is not None

When it happens

Trigger: Passing a custom or exotic PreTrainedConfig lacking num_attention_heads (e.g. some vision/multi-modal backbones or hand-built configs) into the continuous-batching generate path; a config class that names the field differently (n_heads, num_heads).

Common situations: Wiring new architectures into continuous batching; loading models whose attention config lives on sub-configs (text_config) and the outer config is passed instead.

Related errors

• head_dim or (hidden_size and num_attention_heads) could not…
• Invalid group type
• Number of key value heads
• Block size must be at least
• Failed to allocate blocks for request

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2f8a8fe9d2b7f4bc. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/generation/continuous_batching/cache.py:39

from ...generation.configuration_utils import ContinuousBatchingConfig
from ...utils.generic import is_flash_attention_requested
from .cache_manager import BlockManager, CacheAllocator, FullAttentionCacheAllocator, SlidingAttentionCacheAllocator
from .distributed import DistributedHelper
from .initialization import resolve_max_memory_percent
from .requests import RequestState, RequestStatus, get_device_and_memory_breakdown, logger


def find_num_kv_heads(config: PreTrainedConfig) -> int:
    """Finds the number of key-value heads for the given config."""
    # If the model supports GQA, we leverage it by using the num_key_value_heads attribute
    kv_heads = getattr(config, "num_key_value_heads", None)
    if kv_heads is not None:
        return kv_heads
    # Otherwise, the number of KV heads is the same as the number of attention heads
    kv_heads = getattr(config, "num_attention_heads", None)
    if kv_heads is not None:
        return kv_heads
    raise ValueError(f"num_key_value_heads or num_attention_heads could not be found in the config:\n{config}")


def find_head_dim(config: PreTrainedConfig) -> int:
    """Finds the head dimension for the given config."""
    # If the model has the head_dim attribute, there is nothing to do but return it
    head_dim = getattr(config, "head_dim", None)
    if head_dim is not None:
        return head_dim
    # If it is missing, we may reconstruct it from the hidden size and the number of attention heads
    hidden_size = getattr(config, "hidden_size", None)
    num_attention_heads = getattr(config, "num_attention_heads", None)
    if hidden_size is not None and num_attention_heads is not None:
        return hidden_size // num_attention_heads
    raise ValueError(f"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\n{config}")


def group_layers_by_attn_type(config: PreTrainedConfig) -> tuple[list[list[int]], list[str]]:
    """

View on GitHub (pinned to a597f97485)