{"record":{"id":"9319f9d88c085453","repo":"Comfy-Org/ComfyUI","slug":"hidden-size-params-hidden-size-must-be-divisible-9319f9","errorCode":null,"errorMessage":"Hidden size {params.hidden_size} must be divisible by num_heads {params.num_heads}","messagePattern":"Hidden size (.+?) must be divisible by num_heads (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/ldm/flux/model.py","lineNumber":77,"sourceCode":"\n    return result\n\n\nclass Flux(nn.Module):\n    \"\"\"\n    Transformer model for flow matching on sequences.\n    \"\"\"\n\n    def __init__(self, image_model=None, final_layer=True, dtype=None, device=None, operations=None, **kwargs):\n        super().__init__()\n        self.dtype = dtype\n        params = FluxParams(**kwargs)\n        self.params = params\n        self.patch_size = params.patch_size\n        self.in_channels = params.in_channels * params.patch_size * params.patch_size\n        self.out_channels = params.out_channels * params.patch_size * params.patch_size\n        if params.hidden_size % params.num_heads != 0:\n            raise ValueError(\n                f\"Hidden size {params.hidden_size} must be divisible by num_heads {params.num_heads}\"\n            )\n        pe_dim = params.hidden_size // params.num_heads\n        if sum(params.axes_dim) != pe_dim:\n            raise ValueError(f\"Got {params.axes_dim} but expected positional dim {pe_dim}\")\n        self.hidden_size = params.hidden_size\n        self.num_heads = params.num_heads\n        self.pe_embedder = EmbedND(dim=pe_dim, theta=params.theta, axes_dim=params.axes_dim)\n        self.img_in = operations.Linear(self.in_channels, self.hidden_size, bias=params.ops_bias, dtype=dtype, device=device)\n        self.time_in = MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size, bias=params.ops_bias, dtype=dtype, device=device, operations=operations)\n        if params.vec_in_dim is not None:\n            self.vector_in = MLPEmbedder(params.vec_in_dim, self.hidden_size, dtype=dtype, device=device, operations=operations)\n        else:\n            self.vector_in = None\n\n        self.guidance_in = (\n            MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size, bias=params.ops_bias, dtype=dtype, device=device, operations=operations) if params.guidance_embed else nn.Identity()\n        )","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/ldm/flux/model.py#L59-L95","documentation":"Flux's transformer __init__ derives the per-head dimension as hidden_size // num_heads and requires hidden_size to be exactly divisible by num_heads. A non-divisible pair leaves remainder channels that cannot be split across heads, so construction raises ValueError. Both values come from FluxParams (checkpoint config or Flux constructor kwargs).","triggerScenarios":"Creating Flux(params...) with a custom config where hidden_size is not a multiple of num_heads (e.g. hidden_size=1152 with num_heads=16 is fine, but 1150/16 or 3072/28 is not); passing partially-overridden kwargs that break the pairing.","commonSituations":"Custom experiments that resize hidden_size without adjusting num_heads; merging config dicts where one field is updated and the other left stale; community checkpoints with nonstandard head counts.","solutions":["Choose num_heads that divides hidden_size exactly (e.g. for 3072: 24, 16, 12, 8; for 1152: 18, 16, 12).","Restore the stock Flux config values (hidden_size 3072, num_heads 24 for dev/schnell base layers) instead of mixing custom values.","If a checkpoint truly has a non-divisible pair, its attention layout is nonstandard and unsupported here — do not try to pad hidden_size to force it."],"exampleFix":"# before\nFlux(hidden_size=3072, num_heads=28, ...)\n\n# after\nFlux(hidden_size=3072, num_heads=24, ...)","handlingStrategy":"validation","validationCode":"assert params.hidden_size % params.num_heads == 0, \"hidden_size must be divisible by num_heads\"","typeGuard":"def valid_flux_head_config(hidden_size: int, num_heads: int) -> bool:\n    return num_heads > 0 and hidden_size % num_heads == 0","tryCatchPattern":null,"preventionTips":["Treat hidden_size and num_heads as a coupled pair in configs.","Start from the stock Flux params and change one dimension at a time, checking divisibility."],"tags":["flux","config","attention","model-construction"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}