{"record":{"id":"33dc48ffc9c71e3d","repo":"hpcaitech/Open-Sora","slug":"hidden-size-config-hidden-size-must-be-divisible","errorCode":null,"errorMessage":"Hidden size {config.hidden_size} must be divisible by num_heads {config.num_heads}","messagePattern":"Hidden size (.+?) must be divisible by num_heads (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"opensora/models/mmdit/model.py","lineNumber":81,"sourceCode":"        return getattr(self, attribute_name, default)\n\n    def __contains__(self, attribute_name):\n        return hasattr(self, attribute_name)\n\n\nclass MMDiTModel(nn.Module):\n    config_class = MMDiTConfig\n\n    def __init__(self, config: MMDiTConfig):\n        super().__init__()\n\n        self.config = config\n        self.in_channels = config.in_channels\n        self.out_channels = self.in_channels\n        self.patch_size = config.patch_size\n\n        if config.hidden_size % config.num_heads != 0:\n            raise ValueError(\n                f\"Hidden size {config.hidden_size} must be divisible by num_heads {config.num_heads}\"\n            )\n\n        pe_dim = config.hidden_size // config.num_heads\n        if sum(config.axes_dim) != pe_dim:\n            raise ValueError(\n                f\"Got {config.axes_dim} but expected positional dim {pe_dim}\"\n            )\n\n        self.hidden_size = config.hidden_size\n        self.num_heads = config.num_heads\n        pe_embedder_cls = LigerEmbedND if config.use_liger_rope else EmbedND\n        self.pe_embedder = pe_embedder_cls(\n            dim=pe_dim, theta=config.theta, axes_dim=config.axes_dim\n        )\n\n        self.img_in = nn.Linear(self.in_channels, self.hidden_size, bias=True)\n        self.time_in = MLPEmbedder(in_dim=256, hidden_dim=self.hidden_size)","sourceCodeStart":63,"sourceCodeEnd":99,"githubUrl":"https://github.com/hpcaitech/Open-Sora/blob/7ad6a96a135feb81f755c84fb391818718f6beb2/opensora/models/mmdit/model.py#L63-L99","documentation":"The MMDiT model computes per-head positional-embedding dimension as hidden_size // num_heads; this requires hidden_size to be exactly divisible by num_heads. The check runs in __init__ so a misconfigured model fails fast instead of producing ragged heads later.","triggerScenarios":"Building the MMDiT with config values where config.hidden_size % config.num_heads != 0, e.g. hidden_size=3072 with num_heads=25, or a typo'd num_heads in the model config.","commonSituations":"Editing num_heads to tune attention cost without adjusting hidden_size; loading community/config variants with inconsistent values; porting configs between model sizes.","solutions":["Pick num_heads that divides hidden_size evenly (e.g. for 3072 use 24 heads → head_dim 128)","Or adjust hidden_size to the nearest multiple of num_heads","Validate the pair before constructing the model (see validation snippet)"],"exampleFix":"# before\nmodel = MMDit(config)  # hidden_size=3072, num_heads=25\n# after\nconfig.num_heads = 24  # 3072 % 24 == 0\nmodel = MMDit(config)","handlingStrategy":"validation","validationCode":"assert config.hidden_size % config.num_heads == 0, f\"{config.hidden_size} not divisible by {config.num_heads}\"","typeGuard":"def is_valid_head_config(hidden_size: int, num_heads: int) -> bool:\n    return num_heads > 0 and hidden_size % num_heads == 0","tryCatchPattern":null,"preventionTips":["Validate hidden_size/num_heads pairs when loading model configs","Prefer power-of-two head counts","Add a config linter for transformer construction"],"tags":["transformer","mmdit","config","attention"],"backgroundTag":"hidden-size-not-divisible-by-heads","analyzedSha":"7ad6a96a135feb81f755c84fb391818718f6beb2","analyzedAt":"2026-08-28T16:58:37.171Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}