{"record":{"id":"2f8a8fe9d2b7f4bc","repo":"huggingface/transformers","slug":"num-key-value-heads-or-num-attention-heads-could-n","errorCode":null,"errorMessage":"num_key_value_heads or num_attention_heads could not be found in the config:\n{}","messagePattern":"num_key_value_heads or num_attention_heads could not be found in the config:\n(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":39,"sourceCode":"from ...generation.configuration_utils import ContinuousBatchingConfig\nfrom ...utils.generic import is_flash_attention_requested\nfrom .cache_manager import BlockManager, CacheAllocator, FullAttentionCacheAllocator, SlidingAttentionCacheAllocator\nfrom .distributed import DistributedHelper\nfrom .initialization import resolve_max_memory_percent\nfrom .requests import RequestState, RequestStatus, get_device_and_memory_breakdown, logger\n\n\ndef find_num_kv_heads(config: PreTrainedConfig) -> int:\n    \"\"\"Finds the number of key-value heads for the given config.\"\"\"\n    # If the model supports GQA, we leverage it by using the num_key_value_heads attribute\n    kv_heads = getattr(config, \"num_key_value_heads\", None)\n    if kv_heads is not None:\n        return kv_heads\n    # Otherwise, the number of KV heads is the same as the number of attention heads\n    kv_heads = getattr(config, \"num_attention_heads\", None)\n    if kv_heads is not None:\n        return kv_heads\n    raise ValueError(f\"num_key_value_heads or num_attention_heads could not be found in the config:\\n{config}\")\n\n\ndef find_head_dim(config: PreTrainedConfig) -> int:\n    \"\"\"Finds the head dimension for the given config.\"\"\"\n    # If the model has the head_dim attribute, there is nothing to do but return it\n    head_dim = getattr(config, \"head_dim\", None)\n    if head_dim is not None:\n        return head_dim\n    # If it is missing, we may reconstruct it from the hidden size and the number of attention heads\n    hidden_size = getattr(config, \"hidden_size\", None)\n    num_attention_heads = getattr(config, \"num_attention_heads\", None)\n    if hidden_size is not None and num_attention_heads is not None:\n        return hidden_size // num_attention_heads\n    raise ValueError(f\"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\\n{config}\")\n\n\ndef group_layers_by_attn_type(config: PreTrainedConfig) -> tuple[list[list[int]], list[str]]:\n    \"\"\"","sourceCodeStart":21,"sourceCodeEnd":57,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L21-L57","documentation":"ValueError from find_num_kv_heads() in the continuous-batching (vLLM-style) cache: it tries config.num_key_value_heads first (GQA models), then falls back to config.num_attention_heads; if both attributes are missing from the model config, the KV cache cannot size its tensors and the error is raised with the config dump appended.","triggerScenarios":"Passing a custom or exotic PreTrainedConfig lacking num_attention_heads (e.g. some vision/multi-modal backbones or hand-built configs) into the continuous-batching generate path; a config class that names the field differently (n_heads, num_heads).","commonSituations":"Wiring new architectures into continuous batching; loading models whose attention config lives on sub-configs (text_config) and the outer config is passed instead.","solutions":["Ensure the config passed to the cache exposes num_attention_heads (standard text models) or num_key_value_heads (GQA)","For multi-modal models, pass the language-model sub-config (e.g. config.text_config) rather than the top-level config","Set the attribute explicitly before init: config.num_attention_heads = config.n_heads if your config uses a different name"],"exampleFix":"# before\ncache = Cache(config)  # top-level multimodal config, missing num_attention_heads\n# after\ncache = Cache(config.text_config)  # or: config.num_attention_heads = 32 before init","handlingStrategy":"type-guard","validationCode":"assert getattr(config, 'num_key_value_heads', None) or getattr(config, 'num_attention_heads', None), \\\n    'config must define num_attention_heads or num_key_value_heads'","typeGuard":"def has_kv_head_info(config) -> bool:\n    return getattr(config, 'num_key_value_heads', None) is not None or getattr(config, 'num_attention_heads', None) is not None","tryCatchPattern":null,"preventionTips":["Pass the text/language sub-config for multimodal models","Normalize non-standard names (n_heads -> num_attention_heads) before cache init"],"tags":["continuous-batching","kv-cache","model-config","gqa"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}