sgl-project/sglang · critical · ValueError

Hidden size {self.hidden_size} must be divisible by num_atte

Error message

Hidden size {self.hidden_size} must be divisible by num_attention_heads {self.num_attention_heads}

What it means

Transformer hidden size must factor evenly into attention heads because per-head dimension is hidden_size // num_attention_heads. The JoyImage model checks this at construction time before building head-dependent modules (qkv projections, norms, RoPE).

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/joy_image.py:380

        quant_config: Optional[QuantizationConfig] = None,
    ) -> None:
        super().__init__(
            config=config,
            hf_config=hf_config,
        )
        self.in_channels = config.in_channels
        self.out_channels = config.out_channels or config.in_channels
        self.patch_size = config.patch_size
        self.hidden_size = config.hidden_size
        self.num_attention_heads = config.num_attention_heads
        self.rope_dim_list = config.rope_dim_list
        self.mm_double_blocks_depth = config.mm_double_blocks_depth
        self.rope_theta = config.rope_theta
        self.quant_config = quant_config
        self.num_channels_latents = self.out_channels

        if self.hidden_size % self.num_attention_heads != 0:
            raise ValueError(
                f"Hidden size {self.hidden_size} must be divisible by num_attention_heads {self.num_attention_heads}"
            )

        # Image projection (patch embedding)
        self.img_in = nn.Conv3d(
            self.in_channels,
            self.hidden_size,
            kernel_size=self.patch_size,
            stride=self.patch_size,
        )

        # Condition embedding
        self.condition_embedder = WanTimeTextImageEmbedding(
            dim=self.hidden_size,
            time_freq_dim=config.freq_dim,
            text_embed_dim=config.text_states_dim,
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Pick num_attention_heads that divides hidden_size (e.g. hidden 1280 -> heads 20 gives head_dim 64)
2. Fix hidden_size so it is a multiple of the intended head count and head_dim
3. Validate config before constructing the model in your loading script

Example fix

// before
config.num_attention_heads = 24   # hidden_size=1280 -> 1280%24 != 0

// after
config.num_attention_heads = 20   # head_dim = 64
assert config.hidden_size % config.num_attention_heads == 0

def check_joyimage_config(cfg):\n    assert cfg.hidden_size % cfg.num_attention_heads == 0, (\n        f'{cfg.hidden_size=} not divisible by {cfg.num_attention_heads=}')

When it happens

Trigger: Instantiating JoyImage with a config where config.hidden_size % config.num_attention_heads != 0, e.g. hidden_size=1280 with num_attention_heads=24 (head_dim would be fractional).

Common situations: Editing a config.json to shrink/enlarge the model for experiments; converting a checkpoint with non-standard head counts; typos in custom configs.

Related errors

• num_heads ({self.num_heads}) must be divisible by ulysses_de
• Unsupported qk_norm: {qk_norm}
• LingBotWorld requires cross_attn_norm=True
• {rope_type=} not supported. Choose between 'interleaved' and
• Modality {modality} is not supported. Supported modalities a

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2ec393d604023e9b. Report an issue: GitHub.