huggingface/transformers · error · ValueError

head_dim or (hidden_size and num_attention_heads) could not

Error message

head_dim or (hidden_size and num_attention_heads) could not be found in the config:
{}

What it means

ValueError from find_head_dim() in the continuous-batching cache: head dimension is resolved via config.head_dim, or derived as hidden_size // num_attention_heads; if none of these attribute pairs exist, KV cache block tensors cannot be shaped and the error is raised with the config printed. It mirrors find_num_kv_heads and fires on configs that deviate from the standard attention attribute names.

Source

Thrown at src/transformers/generation/continuous_batching/cache.py:53

    # Otherwise, the number of KV heads is the same as the number of attention heads
    kv_heads = getattr(config, "num_attention_heads", None)
    if kv_heads is not None:
        return kv_heads
    raise ValueError(f"num_key_value_heads or num_attention_heads could not be found in the config:\n{config}")


def find_head_dim(config: PreTrainedConfig) -> int:
    """Finds the head dimension for the given config."""
    # If the model has the head_dim attribute, there is nothing to do but return it
    head_dim = getattr(config, "head_dim", None)
    if head_dim is not None:
        return head_dim
    # If it is missing, we may reconstruct it from the hidden size and the number of attention heads
    hidden_size = getattr(config, "hidden_size", None)
    num_attention_heads = getattr(config, "num_attention_heads", None)
    if hidden_size is not None and num_attention_heads is not None:
        return hidden_size // num_attention_heads
    raise ValueError(f"head_dim or (hidden_size and num_attention_heads) could not be found in the config:\n{config}")


def group_layers_by_attn_type(config: PreTrainedConfig) -> tuple[list[list[int]], list[str]]:
    """
    Group layers depending on the attention mix, according to VLLM's hybrid allocator rules:
        - Layers in each group need to have the same type of attention
        - All groups have the same number of layers

    For a model with the following layer types: ["sliding", "full", "full", "sliding", "full", "full", "full", "full"]
    We would get four groups: [0, 3], [1, 2], [4,5] and [6,7].
    """
    # If the config has no layer_type attribute, it means all layers are the same attention type
    layer_types = getattr(config, "layer_types", None)
    if layer_types is None:
        attn_type = "sliding_attention" if getattr(config, "sliding_window", None) is not None else "full_attention"
        layer_types = [attn_type for _ in range(config.num_hidden_layers)]

    # We then count the number of layers of each type

View on GitHub (pinned to a597f97485)

Solutions

1. Set config.head_dim explicitly if the model's head dim is known (e.g. 128) — this takes priority
2. Or ensure both hidden_size and num_attention_heads are present so head_dim = hidden_size // num_attention_heads
3. For multimodal models, pass the text sub-config instead of the top-level one

Example fix

# before
cache = Cache(config)  # ValueError: head_dim ... could not be found
# after
config.head_dim = 128
cache = Cache(config)

if getattr(config, 'head_dim', None) is None:
    hs, nah = getattr(config, 'hidden_size', None), getattr(config, 'num_attention_heads', None)
    assert hs and nah, 'need head_dim or hidden_size+num_attention_heads'
    config.head_dim = hs // nah

def has_head_dim_info(config) -> bool:
    return getattr(config, 'head_dim', None) is not None or (
        getattr(config, 'hidden_size', None) is not None and getattr(config, 'num_attention_heads', None) is not None)

When it happens

Trigger: Custom configs with per-layer head dims but no top-level head_dim/hidden_size; MoE/multimodal wrappers where the outer config lacks hidden_size; deriving configs programmatically and dropping attributes.

Common situations: New architecture onboarding to continuous batching; passing the wrong (outer) config object; configs where hidden_size lives under text_config.

Related errors

• num_key_value_heads or num_attention_heads could not be foun
• Invalid group type: {}
• Block size must be at least {}, but got {}
• Number of key value heads {} must be divisible by tensor par
• Failed to allocate {} blocks for request {}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/75f6434718b8344c. Report an issue: GitHub.