Comfy-Org/ComfyUI · error · ValueError

{name} inner dimension {inner_dim} is not divisible by head

Error message

{name} inner dimension {inner_dim} is not divisible by head dimension {dim_head}

What it means

Raised by _heads_from_dim when reshaping GQA key/value tensors: the tensor's last dimension must be divisible by dim_head so the number of KV heads can be inferred. With grouped-query attention the K/V inner dims may differ from Q's, but each must still be an exact multiple of the head dimension, otherwise the reshape is mathematically impossible and this ValueError fires with the offending name ('Key' or 'Value').

Source

Thrown at comfy/ldm/modules/attention.py:100

        return None

    if FORCE_UPCAST_ATTENTION_DTYPE is not None and current_dtype in FORCE_UPCAST_ATTENTION_DTYPE:
        return FORCE_UPCAST_ATTENTION_DTYPE[current_dtype]
    return attn_precision

def exists(val):
    return val is not None


def default(val, d):
    if exists(val):
        return val
    return d

def _heads_from_dim(tensor, dim_head, name):
    inner_dim = tensor.shape[-1]
    if inner_dim % dim_head != 0:
        raise ValueError(f"{name} inner dimension {inner_dim} is not divisible by head dimension {dim_head}")
    return inner_dim // dim_head

def _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, enable_gqa=False, expand_kv=True):
    q = q.unsqueeze(3).reshape(b, -1, heads, dim_head)
    if enable_gqa:
        key_heads = _heads_from_dim(k, dim_head, "Key")
        value_heads = _heads_from_dim(v, dim_head, "Value")
    else:
        key_heads = heads
        value_heads = heads
    k = k.unsqueeze(3).reshape(b, -1, key_heads, dim_head)
    v = v.unsqueeze(3).reshape(b, -1, value_heads, dim_head)
    if enable_gqa and expand_kv:
        k, v = comfy.ops.repeat_kv_for_gqa(k, v, heads, -2)
    return q, k, v


# feedforward

View on GitHub (pinned to 1c6d8d45b3)

Solutions

Check the model config's head dimension/num_kv_heads against the checkpoint's KV projection shapes
If loading quantized checkpoints, ensure dim_head is derived from the unquantized logical shape, not the packed weight shape
Print q/k/v shapes and dim_head before the reshape to find which tensor is inconsistent

Example fix

# before
heads = _heads_from_dim(k, dim_head=128, name="Key")  # k.shape[-1] == 96 -> raises
# after
dim_head = model_config["head_dim"]  # e.g. 32, divides 96
heads = _heads_from_dim(k, dim_head=dim_head, name="Key")

Defensive patterns

Strategy: validation

Validate before calling

assert q.shape[-1] % dim_head == 0, (q.shape, dim_head)
if enable_gqa:
    assert k.shape[-1] % dim_head == 0, (k.shape, dim_head)
    assert v.shape[-1] % dim_head == 0, (v.shape, dim_head)

Type guard

def kv_heads_valid(k: 'torch.Tensor', dim_head: int) -> bool:
    return k.shape[-1] % dim_head == 0 and k.shape[-1] // dim_head >= 1

Prevention

Derive dim_head from the checkpoint config, not from a packed/quantized weight shape
Log q/k/v shapes and dim_head once at model load for GQA models

When it happens

Trigger: Calling _reshape_qkv_to_heads(..., enable_gqa=True) with k or v tensors whose shape[-1] % dim_head != 0 — e.g. a model checkpoint whose KV projection width doesn't match the configured head size, or a wrong dim_head passed by model glue code.

Common situations: Loading a GQA model with a mismatched config (head_dim inferred from one layer but applied to another); partial/quantized checkpoints that halve KV widths (e.g. 4-bit formats changing second-dim shapes); custom attention patches passing the wrong dim_head.

Related errors

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/3d964c77a5f68634. Report an issue: GitHub.