{"record":{"id":"764662d242b7d473","repo":"sgl-project/sglang","slug":"hidden-size-hidden-size-must-be-divisible-by-num","errorCode":null,"errorMessage":"Hidden size {hidden_size} must be divisible by num_heads {num_heads}","messagePattern":"Hidden size (.+?) must be divisible by num_heads (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/hunyuan3d.py","lineNumber":522,"sourceCode":"\n        self.in_channels = in_channels\n        self.context_in_dim = context_in_dim\n        self.hidden_size = hidden_size\n        self.mlp_ratio = mlp_ratio\n        self.num_heads = num_heads\n        self.num_attention_heads = num_heads\n        self.depth = depth\n        self.depth_single_blocks = depth_single_blocks\n        self.axes_dim = axes_dim\n        self.theta = theta\n        self.qkv_bias = qkv_bias\n        self.time_factor = time_factor\n        self.out_channels = self.in_channels\n        self.num_channels_latents = self.in_channels\n        self.guidance_embed = guidance_embed\n\n        if hidden_size % num_heads != 0:\n            raise ValueError(\n                f\"Hidden size {hidden_size} must be divisible by num_heads {num_heads}\"\n            )\n        pe_dim = hidden_size // num_heads\n        if sum(axes_dim) != pe_dim:\n            raise ValueError(f\"Got {axes_dim} but expected positional dim {pe_dim}\")\n        self.latent_in = nn.Linear(self.in_channels, self.hidden_size, bias=True)\n        self.time_in = _FluxMLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)\n        self.cond_in = nn.Linear(context_in_dim, self.hidden_size)\n        self.guidance_in = (\n            _FluxMLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)\n            if guidance_embed\n            else nn.Identity()\n        )\n\n        self.double_blocks = nn.ModuleList(\n            [\n                _FluxDoubleStreamBlock(\n                    self.hidden_size,","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/hunyuan3d.py#L504-L540","documentation":"The Hunyuan3D DiT requires hidden_size to be divisible by num_heads so each head gets hidden_size//num_heads channels for its axial positional encoding. This ValueError is thrown in __init__ when the divisibility check fails.","triggerScenarios":"Constructing the Hunyuan3D transformer with a hidden_size/num_heads pair where hidden_size % num_heads != 0, e.g. hidden_size=1152 with num_heads=14 (1152/14 is not an integer).","commonSituations":"Overriding model width or head count for ablations/pruning without keeping divisibility; merging a config from a variant model with different head counts.","solutions":["Pick num_heads that divides hidden_size exactly (e.g. 1152 → 18 heads of 64, or 16 heads of 72)","Adjust hidden_size to a multiple of num_heads if you control the width","Restore the original pretrained config values for hidden_size and num_heads"],"exampleFix":"# before\nTransformer(hidden_size=1152, num_heads=14, ...)\n\n# after\nTransformer(hidden_size=1152, num_heads=18, ...)","handlingStrategy":"validation","validationCode":"assert hidden_size % num_heads == 0, f'{hidden_size} % {num_heads} != 0'","typeGuard":"def heads_divide_hidden(hidden_size: int, num_heads: int) -> bool:\n    return hidden_size % num_heads == 0","tryCatchPattern":null,"preventionTips":["Validate head/width pairs in config sanity checks","When overriding num_heads for experiments, assert divisibility in the same commit","Add a config linter for transformer hyperparameters"],"tags":["config","validation","transformer","diffusion"],"backgroundTag":"hidden-size-not-divisible-by-heads","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}