{"record":{"id":"75f6434718b8344c","repo":"huggingface/transformers","slug":"head-dim-or-hidden-size-and-num-attention-heads","errorCode":null,"errorMessage":"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\n{}","messagePattern":"head_dim or \\(hidden_size and num_attention_heads\\) could not be found in the config:\n(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/cache.py","lineNumber":53,"sourceCode":"    # Otherwise, the number of KV heads is the same as the number of attention heads\n    kv_heads = getattr(config, \"num_attention_heads\", None)\n    if kv_heads is not None:\n        return kv_heads\n    raise ValueError(f\"num_key_value_heads or num_attention_heads could not be found in the config:\\n{config}\")\n\n\ndef find_head_dim(config: PreTrainedConfig) -> int:\n    \"\"\"Finds the head dimension for the given config.\"\"\"\n    # If the model has the head_dim attribute, there is nothing to do but return it\n    head_dim = getattr(config, \"head_dim\", None)\n    if head_dim is not None:\n        return head_dim\n    # If it is missing, we may reconstruct it from the hidden size and the number of attention heads\n    hidden_size = getattr(config, \"hidden_size\", None)\n    num_attention_heads = getattr(config, \"num_attention_heads\", None)\n    if hidden_size is not None and num_attention_heads is not None:\n        return hidden_size // num_attention_heads\n    raise ValueError(f\"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\\n{config}\")\n\n\ndef group_layers_by_attn_type(config: PreTrainedConfig) -> tuple[list[list[int]], list[str]]:\n    \"\"\"\n    Group layers depending on the attention mix, according to VLLM's hybrid allocator rules:\n        - Layers in each group need to have the same type of attention\n        - All groups have the same number of layers\n\n    For a model with the following layer types: [\"sliding\", \"full\", \"full\", \"sliding\", \"full\", \"full\", \"full\", \"full\"]\n    We would get four groups: [0, 3], [1, 2], [4,5] and [6,7].\n    \"\"\"\n    # If the config has no layer_type attribute, it means all layers are the same attention type\n    layer_types = getattr(config, \"layer_types\", None)\n    if layer_types is None:\n        attn_type = \"sliding_attention\" if getattr(config, \"sliding_window\", None) is not None else \"full_attention\"\n        layer_types = [attn_type for _ in range(config.num_hidden_layers)]\n\n    # We then count the number of layers of each type","sourceCodeStart":35,"sourceCodeEnd":71,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/cache.py#L35-L71","documentation":"ValueError from find_head_dim() in the continuous-batching cache: head dimension is resolved via config.head_dim, or derived as hidden_size // num_attention_heads; if none of these attribute pairs exist, KV cache block tensors cannot be shaped and the error is raised with the config printed. It mirrors find_num_kv_heads and fires on configs that deviate from the standard attention attribute names.","triggerScenarios":"Custom configs with per-layer head dims but no top-level head_dim/hidden_size; MoE/multimodal wrappers where the outer config lacks hidden_size; deriving configs programmatically and dropping attributes.","commonSituations":"New architecture onboarding to continuous batching; passing the wrong (outer) config object; configs where hidden_size lives under text_config.","solutions":["Set config.head_dim explicitly if the model's head dim is known (e.g. 128) — this takes priority","Or ensure both hidden_size and num_attention_heads are present so head_dim = hidden_size // num_attention_heads","For multimodal models, pass the text sub-config instead of the top-level one"],"exampleFix":"# before\ncache = Cache(config)  # ValueError: head_dim ... could not be found\n# after\nconfig.head_dim = 128\ncache = Cache(config)","handlingStrategy":"type-guard","validationCode":"if getattr(config, 'head_dim', None) is None:\n    hs, nah = getattr(config, 'hidden_size', None), getattr(config, 'num_attention_heads', None)\n    assert hs and nah, 'need head_dim or hidden_size+num_attention_heads'\n    config.head_dim = hs // nah","typeGuard":"def has_head_dim_info(config) -> bool:\n    return getattr(config, 'head_dim', None) is not None or (\n        getattr(config, 'hidden_size', None) is not None and getattr(config, 'num_attention_heads', None) is not None)","tryCatchPattern":null,"preventionTips":["Set config.head_dim explicitly when porting models","Ensure hidden_size is divisible by num_attention_heads before deriving"],"tags":["continuous-batching","kv-cache","model-config","head-dim"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}