{"record":{"id":"bafbfce8399094c3","repo":"opendatalab/MinerU","slug":"the-hidden-size-config-hidden-size-is-not-a-mu","errorCode":null,"errorMessage":"The hidden size ({config.hidden_size}) is not a multiple of the number of attention heads ({config.num_attention_heads})","messagePattern":"The hidden size \\((.+?)\\) is not a multiple of the number of attention heads \\((.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mineru/model/layout/pp_doclayoutv2.py","lineNumber":394,"sourceCode":"        return torch.cat([relative_coordinates, relative_dim], dim=-1)\n\n    def get_position_embedding(self, x: torch.Tensor, scale: float = 100.0) -> torch.Tensor:\n        embedding = (x * scale).unsqueeze(-1) * self.inv_freq\n        return torch.cat((embedding.sin(), embedding.cos()), dim=-1).flatten(start_dim=-2).to(x.dtype)\n\n    def forward(self, source_boxes: torch.Tensor, target_boxes: Optional[torch.Tensor] = None) -> torch.Tensor:\n        target_boxes = source_boxes if target_boxes is None else target_boxes\n        with torch.no_grad():\n            relative_encoding = self.box_relative_encoding(source_boxes, target_boxes)\n            position_embedding = self.get_position_embedding(relative_encoding, self.scale).permute(0, 3, 1, 2)\n        return self.pos_proj(position_embedding)\n\n\nclass PPDocLayoutV2ReadingOrderSelfAttention(nn.Module):\n    def __init__(self, config: PPDocLayoutV2ReadingOrderConfig):\n        super().__init__()\n        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, \"embedding_size\"):\n            raise ValueError(\n                f\"The hidden size ({config.hidden_size}) is not a multiple of the number of attention heads \"\n                f\"({config.num_attention_heads})\"\n            )\n\n        self.num_attention_heads = config.num_attention_heads\n        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)\n        self.all_head_size = self.num_attention_heads * self.attention_head_size\n        self.query = nn.Linear(config.hidden_size, self.all_head_size)\n        self.key = nn.Linear(config.hidden_size, self.all_head_size)\n        self.value = nn.Linear(config.hidden_size, self.all_head_size)\n        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)\n        self.has_relative_attention_bias = config.has_relative_attention_bias\n        self.has_spatial_attention_bias = config.has_spatial_attention_bias\n\n    @staticmethod\n    def cogview_attention(attention_scores: torch.Tensor, alpha: float = 32.0) -> torch.Tensor:\n        scaled_attention_scores = attention_scores / alpha\n        max_value = scaled_attention_scores.amax(dim=-1, keepdim=True)","sourceCodeStart":376,"sourceCodeEnd":412,"githubUrl":"https://github.com/opendatalab/MinerU/blob/4fe4bde114a23ee5dd637eae99b767f4669bf58c/mineru/model/layout/pp_doclayoutv2.py#L376-L412","documentation":"ValueError raised in PPDocLayoutV2ReadingOrderSelfAttention.__init__ when config.hidden_size is not divisible by config.num_attention_heads (and no 'embedding_size' attribute exists on the config). Multi-head attention splits the hidden size across heads, so a non-divisible pair cannot produce equal head sizes and construction fails immediately.","triggerScenarios":"Loading PP-DocLayoutV2 reading-order weights with a locally modified config (e.g. hidden_size=768, num_attention_heads=10), or constructing PPDocLayoutV2ReadingOrderConfig with custom values that were never validated.","commonSituations":"Hand-tuned configs for experiments; porting a config from another model family; corrupted/partially edited config.json after download; version mismatches between a checkpoint's config and the code's expected fields.","solutions":["Restore the original checkpoint config values for hidden_size and num_attention_heads (do not hand-edit them).","Choose a num_attention_heads that divides hidden_size (e.g. 768 -> 12 heads, 256 -> 8 heads).","Re-download the model/config from the official source to eliminate corruption.","If you intentionally added an 'embedding_size' projection, set config.embedding_size so the check is bypassed as designed."],"exampleFix":"# before\ncfg = PPDocLayoutV2ReadingOrderConfig(hidden_size=768, num_attention_heads=10)\nattn = PPDocLayoutV2ReadingOrderSelfAttention(cfg)  # ValueError\n\n# after\ncfg = PPDocLayoutV2ReadingOrderConfig(hidden_size=768, num_attention_heads=12)\nattn = PPDocLayoutV2ReadingOrderSelfAttention(cfg)","handlingStrategy":"validation","validationCode":"def validate_head_config(hidden_size: int, num_attention_heads: int) -> None:\n    if hidden_size % num_attention_heads != 0:\n        raise ValueError(\n            f'hidden_size {hidden_size} must be divisible by num_attention_heads {num_attention_heads}'\n        )\n\nvalidate_head_config(cfg.hidden_size, cfg.num_attention_heads)\nattn = PPDocLayoutV2ReadingOrderSelfAttention(cfg)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never hand-edit hidden_size or num_attention_heads of a released checkpoint.","Validate divisibility in your config-loading code with a clear message.","Re-download configs from the official source when upgrades change schema."],"tags":["pytorch","deep-learning","config","transformer","layout-model"],"backgroundTag":null,"analyzedSha":"4fe4bde114a23ee5dd637eae99b767f4669bf58c","analyzedAt":"2026-08-14T21:29:18.456Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}