sgl-project/sglang · error · ValueError

Hidden size {hidden_size} must be divisible by num_heads {nu

Error message

Hidden size {hidden_size} must be divisible by num_heads {num_heads}

What it means

The Hunyuan3D DiT requires hidden_size to be divisible by num_heads so each head gets hidden_size//num_heads channels for its axial positional encoding. This ValueError is thrown in __init__ when the divisibility check fails.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/hunyuan3d.py:522

        self.in_channels = in_channels
        self.context_in_dim = context_in_dim
        self.hidden_size = hidden_size
        self.mlp_ratio = mlp_ratio
        self.num_heads = num_heads
        self.num_attention_heads = num_heads
        self.depth = depth
        self.depth_single_blocks = depth_single_blocks
        self.axes_dim = axes_dim
        self.theta = theta
        self.qkv_bias = qkv_bias
        self.time_factor = time_factor
        self.out_channels = self.in_channels
        self.num_channels_latents = self.in_channels
        self.guidance_embed = guidance_embed

        if hidden_size % num_heads != 0:
            raise ValueError(
                f"Hidden size {hidden_size} must be divisible by num_heads {num_heads}"
            )
        pe_dim = hidden_size // num_heads
        if sum(axes_dim) != pe_dim:
            raise ValueError(f"Got {axes_dim} but expected positional dim {pe_dim}")
        self.latent_in = nn.Linear(self.in_channels, self.hidden_size, bias=True)
        self.time_in = _FluxMLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)
        self.cond_in = nn.Linear(context_in_dim, self.hidden_size)
        self.guidance_in = (
            _FluxMLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)
            if guidance_embed
            else nn.Identity()
        )

        self.double_blocks = nn.ModuleList(
            [
                _FluxDoubleStreamBlock(
                    self.hidden_size,

View on GitHub (pinned to 0132848349)

Solutions

1. Pick num_heads that divides hidden_size exactly (e.g. 1152 → 18 heads of 64, or 16 heads of 72)
2. Adjust hidden_size to a multiple of num_heads if you control the width
3. Restore the original pretrained config values for hidden_size and num_heads

Example fix

# before
Transformer(hidden_size=1152, num_heads=14, ...)

# after
Transformer(hidden_size=1152, num_heads=18, ...)

assert hidden_size % num_heads == 0, f'{hidden_size} % {num_heads} != 0'

def heads_divide_hidden(hidden_size: int, num_heads: int) -> bool:
    return hidden_size % num_heads == 0

When it happens

Trigger: Constructing the Hunyuan3D transformer with a hidden_size/num_heads pair where hidden_size % num_heads != 0, e.g. hidden_size=1152 with num_heads=14 (1152/14 is not an integer).

Common situations: Overriding model width or head count for ablations/pruning without keeping divisibility; merging a config from a variant model with different head counts.

Related errors

• unknown qk_norm: {qk_norm}. Should be one of None, 'layer_no
• unknown norm_type {norm_type}
• Unknown history_scale_mode: {history_scale_mode}
• Got {axes_dim} but expected positional dim {pe_dim}
• denoising_strength must be positive

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/764662d242b7d473. Report an issue: GitHub.