{"record":{"id":"2ec393d604023e9b","repo":"sgl-project/sglang","slug":"hidden-size-self-hidden-size-must-be-divisible-b","errorCode":null,"errorMessage":"Hidden size {self.hidden_size} must be divisible by num_attention_heads {self.num_attention_heads}","messagePattern":"Hidden size (.+?) must be divisible by num_attention_heads (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/models/dits/joy_image.py","lineNumber":380,"sourceCode":"        quant_config: Optional[QuantizationConfig] = None,\n    ) -> None:\n        super().__init__(\n            config=config,\n            hf_config=hf_config,\n        )\n        self.in_channels = config.in_channels\n        self.out_channels = config.out_channels or config.in_channels\n        self.patch_size = config.patch_size\n        self.hidden_size = config.hidden_size\n        self.num_attention_heads = config.num_attention_heads\n        self.rope_dim_list = config.rope_dim_list\n        self.mm_double_blocks_depth = config.mm_double_blocks_depth\n        self.rope_theta = config.rope_theta\n        self.quant_config = quant_config\n        self.num_channels_latents = self.out_channels\n\n        if self.hidden_size % self.num_attention_heads != 0:\n            raise ValueError(\n                f\"Hidden size {self.hidden_size} must be divisible by num_attention_heads {self.num_attention_heads}\"\n            )\n\n        # Image projection (patch embedding)\n        self.img_in = nn.Conv3d(\n            self.in_channels,\n            self.hidden_size,\n            kernel_size=self.patch_size,\n            stride=self.patch_size,\n        )\n\n        # Condition embedding\n        self.condition_embedder = WanTimeTextImageEmbedding(\n            dim=self.hidden_size,\n            time_freq_dim=config.freq_dim,\n            text_embed_dim=config.text_states_dim,\n        )\n","sourceCodeStart":362,"sourceCodeEnd":398,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/joy_image.py#L362-L398","documentation":"Transformer hidden size must factor evenly into attention heads because per-head dimension is hidden_size // num_attention_heads. The JoyImage model checks this at construction time before building head-dependent modules (qkv projections, norms, RoPE).","triggerScenarios":"Instantiating JoyImage with a config where config.hidden_size % config.num_attention_heads != 0, e.g. hidden_size=1280 with num_attention_heads=24 (head_dim would be fractional).","commonSituations":"Editing a config.json to shrink/enlarge the model for experiments; converting a checkpoint with non-standard head counts; typos in custom configs.","solutions":["Pick num_attention_heads that divides hidden_size (e.g. hidden 1280 -> heads 20 gives head_dim 64)","Fix hidden_size so it is a multiple of the intended head count and head_dim","Validate config before constructing the model in your loading script"],"exampleFix":"// before\nconfig.num_attention_heads = 24   # hidden_size=1280 -> 1280%24 != 0\n\n// after\nconfig.num_attention_heads = 20   # head_dim = 64\nassert config.hidden_size % config.num_attention_heads == 0","handlingStrategy":"validation","validationCode":"def check_joyimage_config(cfg):\\n    assert cfg.hidden_size % cfg.num_attention_heads == 0, (\\n        f'{cfg.hidden_size=} not divisible by {cfg.num_attention_heads=}')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate configs in a loader before model construction","Prefer head counts that are powers of two","Add a config unit test for every custom variant"],"tags":["config-validation","attention-heads","init-time","joyimage"],"backgroundTag":"model-config-validation-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}