{"record":{"id":"3d964c77a5f68634","repo":"Comfy-Org/ComfyUI","slug":"name-inner-dimension-inner-dim-is-not-divisibl","errorCode":null,"errorMessage":"{name} inner dimension {inner_dim} is not divisible by head dimension {dim_head}","messagePattern":"(.+?) inner dimension (.+?) is not divisible by head dimension (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/modules/attention.py","lineNumber":100,"sourceCode":"        return None\n\n    if FORCE_UPCAST_ATTENTION_DTYPE is not None and current_dtype in FORCE_UPCAST_ATTENTION_DTYPE:\n        return FORCE_UPCAST_ATTENTION_DTYPE[current_dtype]\n    return attn_precision\n\ndef exists(val):\n    return val is not None\n\n\ndef default(val, d):\n    if exists(val):\n        return val\n    return d\n\ndef _heads_from_dim(tensor, dim_head, name):\n    inner_dim = tensor.shape[-1]\n    if inner_dim % dim_head != 0:\n        raise ValueError(f\"{name} inner dimension {inner_dim} is not divisible by head dimension {dim_head}\")\n    return inner_dim // dim_head\n\ndef _reshape_qkv_to_heads(q, k, v, b, heads, dim_head, enable_gqa=False, expand_kv=True):\n    q = q.unsqueeze(3).reshape(b, -1, heads, dim_head)\n    if enable_gqa:\n        key_heads = _heads_from_dim(k, dim_head, \"Key\")\n        value_heads = _heads_from_dim(v, dim_head, \"Value\")\n    else:\n        key_heads = heads\n        value_heads = heads\n    k = k.unsqueeze(3).reshape(b, -1, key_heads, dim_head)\n    v = v.unsqueeze(3).reshape(b, -1, value_heads, dim_head)\n    if enable_gqa and expand_kv:\n        k, v = comfy.ops.repeat_kv_for_gqa(k, v, heads, -2)\n    return q, k, v\n\n\n# feedforward","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/modules/attention.py#L82-L118","documentation":"Raised by _heads_from_dim when reshaping GQA key/value tensors: the tensor's last dimension must be divisible by dim_head so the number of KV heads can be inferred. With grouped-query attention the K/V inner dims may differ from Q's, but each must still be an exact multiple of the head dimension, otherwise the reshape is mathematically impossible and this ValueError fires with the offending name ('Key' or 'Value').","triggerScenarios":"Calling _reshape_qkv_to_heads(..., enable_gqa=True) with k or v tensors whose shape[-1] % dim_head != 0 — e.g. a model checkpoint whose KV projection width doesn't match the configured head size, or a wrong dim_head passed by model glue code.","commonSituations":"Loading a GQA model with a mismatched config (head_dim inferred from one layer but applied to another); partial/quantized checkpoints that halve KV widths (e.g. 4-bit formats changing second-dim shapes); custom attention patches passing the wrong dim_head.","solutions":["Check the model config's head dimension/num_kv_heads against the checkpoint's KV projection shapes","If loading quantized checkpoints, ensure dim_head is derived from the unquantized logical shape, not the packed weight shape","Print q/k/v shapes and dim_head before the reshape to find which tensor is inconsistent"],"exampleFix":"# before\nheads = _heads_from_dim(k, dim_head=128, name=\"Key\")  # k.shape[-1] == 96 -> raises\n# after\ndim_head = model_config[\"head_dim\"]  # e.g. 32, divides 96\nheads = _heads_from_dim(k, dim_head=dim_head, name=\"Key\")","handlingStrategy":"validation","validationCode":"assert q.shape[-1] % dim_head == 0, (q.shape, dim_head)\nif enable_gqa:\n    assert k.shape[-1] % dim_head == 0, (k.shape, dim_head)\n    assert v.shape[-1] % dim_head == 0, (v.shape, dim_head)","typeGuard":"def kv_heads_valid(k: 'torch.Tensor', dim_head: int) -> bool:\n    return k.shape[-1] % dim_head == 0 and k.shape[-1] // dim_head >= 1","tryCatchPattern":null,"preventionTips":["Derive dim_head from the checkpoint config, not from a packed/quantized weight shape","Log q/k/v shapes and dim_head once at model load for GQA models"],"tags":["attention","gqa","shape-mismatch","model-loading"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}