hpcaitech/Open-Sora · error · ValueError

Hidden size {config.hidden_size} must be divisible by num_he

Error message

Hidden size {config.hidden_size} must be divisible by num_heads {config.num_heads}

What it means

The MMDiT model computes per-head positional-embedding dimension as hidden_size // num_heads; this requires hidden_size to be exactly divisible by num_heads. The check runs in __init__ so a misconfigured model fails fast instead of producing ragged heads later.

Source

Thrown at opensora/models/mmdit/model.py:81

        return getattr(self, attribute_name, default)

    def __contains__(self, attribute_name):
        return hasattr(self, attribute_name)


class MMDiTModel(nn.Module):
    config_class = MMDiTConfig

    def __init__(self, config: MMDiTConfig):
        super().__init__()

        self.config = config
        self.in_channels = config.in_channels
        self.out_channels = self.in_channels
        self.patch_size = config.patch_size

        if config.hidden_size % config.num_heads != 0:
            raise ValueError(
                f"Hidden size {config.hidden_size} must be divisible by num_heads {config.num_heads}"
            )

        pe_dim = config.hidden_size // config.num_heads
        if sum(config.axes_dim) != pe_dim:
            raise ValueError(
                f"Got {config.axes_dim} but expected positional dim {pe_dim}"
            )

        self.hidden_size = config.hidden_size
        self.num_heads = config.num_heads
        pe_embedder_cls = LigerEmbedND if config.use_liger_rope else EmbedND
        self.pe_embedder = pe_embedder_cls(
            dim=pe_dim, theta=config.theta, axes_dim=config.axes_dim
        )

        self.img_in = nn.Linear(self.in_channels, self.hidden_size, bias=True)
        self.time_in = MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)

View on GitHub (pinned to 7ad6a96a13)

Solutions

Pick num_heads that divides hidden_size evenly (e.g. for 3072 use 24 heads → head_dim 128)
Or adjust hidden_size to the nearest multiple of num_heads
Validate the pair before constructing the model (see validation snippet)

Example fix

# before
model = MMDit(config)  # hidden_size=3072, num_heads=25
# after
config.num_heads = 24  # 3072 % 24 == 0
model = MMDit(config)

Defensive patterns

Strategy: validation

Validate before calling

assert config.hidden_size % config.num_heads == 0, f"{config.hidden_size} not divisible by {config.num_heads}"

Type guard

def is_valid_head_config(hidden_size: int, num_heads: int) -> bool:
    return num_heads > 0 and hidden_size % num_heads == 0

Prevention

Validate hidden_size/num_heads pairs when loading model configs
Prefer power-of-two head counts
Add a config linter for transformer construction

When it happens

Trigger: Building the MMDiT with config values where config.hidden_size % config.num_heads != 0, e.g. hidden_size=3072 with num_heads=25, or a typo'd num_heads in the model config.

Common situations: Editing num_heads to tune attention cost without adjusting hidden_size; loading community/config variants with inconsistent values; porting configs between model sizes.

Related errors

AI-assisted analysis of hpcaitech/Open-Sora@7ad6a96a13 (2026-08-28). Data as JSON: /api/errors/33dc48ffc9c71e3d. Report an issue: GitHub.