{"record":{"id":"e0c82fc5eba0f89b","repo":"invoke-ai/InvokeAI","slug":"hidden-size-params-hidden-size-must-be-divisible-e0c82f","errorCode":null,"errorMessage":"Hidden size {params.hidden_size} must be divisible by num_heads {params.num_heads}","messagePattern":"Hidden size (.+?) must be divisible by num_heads (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/flux/model.py","lineNumber":54,"sourceCode":"    theta: int\n    qkv_bias: bool\n    guidance_embed: bool\n    out_channels: Optional[int] = None\n\n\nclass Flux(nn.Module):\n    \"\"\"\n    Transformer model for flow matching on sequences.\n    \"\"\"\n\n    def __init__(self, params: FluxParams):\n        super().__init__()\n\n        self.params = params\n        self.in_channels = params.in_channels\n        self.out_channels = params.out_channels or self.in_channels\n        if params.hidden_size % params.num_heads != 0:\n            raise ValueError(f\"Hidden size {params.hidden_size} must be divisible by num_heads {params.num_heads}\")\n        pe_dim = params.hidden_size // params.num_heads\n        if sum(params.axes_dim) != pe_dim:\n            raise ValueError(f\"Got {params.axes_dim} but expected positional dim {pe_dim}\")\n        self.hidden_size = params.hidden_size\n        self.num_heads = params.num_heads\n        self.pe_embedder = EmbedND(dim=pe_dim, theta=params.theta, axes_dim=params.axes_dim)\n        self.img_in = nn.Linear(self.in_channels, self.hidden_size, bias=True)\n        self.time_in = MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)\n        self.vector_in = MLPEmbedder(params.vec_in_dim, self.hidden_size)\n        self.guidance_in = (\n            MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size) if params.guidance_embed else nn.Identity()\n        )\n        self.txt_in = nn.Linear(params.context_in_dim, self.hidden_size)\n\n        self.double_blocks = nn.ModuleList(\n            [\n                DoubleStreamBlock(\n                    self.hidden_size,","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/flux/model.py#L36-L72","documentation":"Flux model __init__ validates that hidden_size is evenly divisible by num_heads before computing per-head dimensions. If hidden_size % num_heads != 0, the multi-head attention head dimension would be fractional, so construction is aborted with a ValueError. This is a model-configuration sanity check mirroring the original FLUX reference implementation.","triggerScenarios":"Constructing Flux (or Flux2) model with FluxParams where hidden_size is not a multiple of num_heads, e.g. copying a config and editing hidden_size manually, or loading a variant whose params dict was modified.","commonSituations":"Hand-edited config JSON/YAML for a custom FLUX variant; porting state dicts from forks with mismatched param blocks; typo in hidden_size (e.g. 3072 vs 3073) while num_heads stays at 24.","solutions":["Set hidden_size to a multiple of num_heads (FLUX default: hidden_size=3072, num_heads=24, giving pe_dim=128)","Use get_flux_transformers_params(variant) from invokeai/backend/flux/util.py instead of hand-rolling FluxParams","If you need a custom width, change num_heads so hidden_size % num_heads == 0 and verify sum(axes_dim) equals hidden_size // num_heads"],"exampleFix":"// before\nparams = FluxParams(in_channels=64, hidden_size=3000, num_heads=24, axes_dim=[128,128,128], ...)\n// after\nparams = FluxParams(in_channels=64, hidden_size=3072, num_heads=24, axes_dim=[128,128,128], ...)","handlingStrategy":"validation","validationCode":"assert params.hidden_size % params.num_heads == 0, f\"hidden_size {params.hidden_size} not divisible by num_heads {params.num_heads}\"","typeGuard":"def is_valid_flux_head_config(params) -> bool:\n    return params.hidden_size % params.num_heads == 0","tryCatchPattern":"try:\n    model = Flux(params)\nexcept ValueError as e:\n    if \"divisible by num_heads\" in str(e):\n        params.num_heads = pick_num_heads_dividing(params.hidden_size)\n        model = Flux(params)\n    else:\n        raise","preventionTips":["Always derive FluxParams from get_flux_transformers_params(variant)","Keep hidden_size and num_heads changed together: hidden_size = num_heads * head_dim","Write a unit test that instantiates every variant's params"],"tags":["flux","config-validation","model-init","transformer"],"backgroundTag":"model-config-validation","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}