{"record":{"id":"047b1347e1fb0fd9","repo":"Comfy-Org/ComfyUI","slug":"dim-x-dim-x-should-be-divisible-by-num-heads-nu","errorCode":null,"errorMessage":"dim_x={dim_x} should be divisible by num_heads={num_heads}","messagePattern":"dim_x=(.+?) should be divisible by num_heads=(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/genmo/joint_model/asymm_models_joint.py","lineNumber":80,"sourceCode":"        update_y: bool = True,\n        out_bias: bool = True,\n        attend_to_padding: bool = False,\n        softmax_scale: Optional[float] = None,\n        device: Optional[torch.device] = None,\n        dtype=None,\n        operations=None,\n    ):\n        super().__init__()\n        self.dim_x = dim_x\n        self.dim_y = dim_y\n        self.num_heads = num_heads\n        self.head_dim = dim_x // num_heads\n        self.attn_drop = attn_drop\n        self.update_y = update_y\n        self.attend_to_padding = attend_to_padding\n        self.softmax_scale = softmax_scale\n        if dim_x % num_heads != 0:\n            raise ValueError(\n                f\"dim_x={dim_x} should be divisible by num_heads={num_heads}\"\n            )\n\n        # Input layers.\n        self.qkv_bias = qkv_bias\n        self.qkv_x = operations.Linear(dim_x, 3 * dim_x, bias=qkv_bias, device=device, dtype=dtype)\n        # Project text features to match visual features (dim_y -> dim_x)\n        self.qkv_y = operations.Linear(dim_y, 3 * dim_x, bias=qkv_bias, device=device, dtype=dtype)\n\n        # Query and key normalization for stability.\n        assert qk_norm\n        self.q_norm_x = operations.RMSNorm(self.head_dim, eps=1e-5, device=device, dtype=dtype)\n        self.k_norm_x = operations.RMSNorm(self.head_dim, eps=1e-5, device=device, dtype=dtype)\n        self.q_norm_y = operations.RMSNorm(self.head_dim, eps=1e-5, device=device, dtype=dtype)\n        self.k_norm_y = operations.RMSNorm(self.head_dim, eps=1e-5, device=device, dtype=dtype)\n\n        # Output layers. y features go back down from dim_x -> dim_y.\n        self.proj_x = operations.Linear(dim_x, dim_x, bias=out_bias, device=device, dtype=dtype)","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/genmo/joint_model/asymm_models_joint.py#L62-L98","documentation":"The Mochi joint attention module splits the visual stream dim_x across num_heads (head_dim = dim_x // num_heads) and requires exact divisibility. A non-divisible pair makes per-head RMSNorm and attention shapes ill-defined, so __init__ raises ValueError. dim_x/dim_y/num_heads come from the Mochi model config.","triggerScenarios":"Constructing AsymmetricJointBlock with dim_x not a multiple of num_heads (e.g. 3072 with 28 heads); a partial config override that changes dim_x (widened model) but keeps the original head count.","commonSituations":"Experimenting with Mochi architecture variants; community checkpoints with modified widths; config typos when hand-writing the Mochi params dict.","solutions":["Pick num_heads that divides dim_x exactly (stock Mochi: dim_x 3072 with 24 heads).","When scaling dim_x, scale num_heads to keep head_dim integral (e.g. 48*64 -> 24/32/48 heads).","Validate dim_x % num_heads == 0 in your loader before building the model."],"exampleFix":"# before\nAsymmetricJointBlock(dim_x=3072, dim_y=1536, num_heads=28, ...)\n\n# after\nAsymmetricJointBlock(dim_x=3072, dim_y=1536, num_heads=24, ...)","handlingStrategy":"validation","validationCode":"assert dim_x % num_heads == 0, f\"dim_x={dim_x} not divisible by num_heads={num_heads}\"","typeGuard":"def valid_mochi_head_config(dim_x: int, num_heads: int) -> bool:\n    return num_heads > 0 and dim_x % num_heads == 0","tryCatchPattern":null,"preventionTips":["Keep the stock Mochi dim_x/num_heads pairing (3072/24) unless you recompute both.","Validate divisibility in loaders for custom widths."],"tags":["mochi","genmo","attention","config"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}