{"record":{"id":"f1188eb8e2cdaef1","repo":"sgl-project/sglang","slug":"embed-dim-must-be-divisible-by-num-heads-got-emb","errorCode":null,"errorMessage":"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads}).","messagePattern":"embed_dim must be divisible by num_heads \\(got `embed_dim`: (.+?) and `num_heads`: (.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/siglip2.py","lineNumber":187,"sourceCode":"\n\nclass Siglip2Attention(nn.Module):\n    \"\"\"Multi-headed attention for Siglip2 using optimized VisionAttention backend.\"\"\"\n\n    def __init__(\n        self,\n        config: Siglip2VisionConfig,\n        quant_config: Optional[QuantizationConfig] = None,\n        prefix: str = \"\",\n    ):\n        super().__init__()\n        self.config = config\n        self.embed_dim = config.hidden_size\n        self.num_heads = config.num_attention_heads\n        self.head_dim = self.embed_dim // self.num_heads\n\n        if self.head_dim * self.num_heads != self.embed_dim:\n            raise ValueError(\n                f\"embed_dim must be divisible by num_heads \"\n                f\"(got `embed_dim`: {self.embed_dim} and `num_heads`:\"\n                f\" {self.num_heads}).\"\n            )\n\n        # Use SGLang's optimized VisionAttention with automatic backend selection\n        self.attn = VisionAttention(\n            embed_dim=self.embed_dim,\n            num_heads=self.num_heads,\n            projection_size=self.embed_dim,\n            use_qkv_parallel=True,\n            dropout=config.attention_dropout,\n            flatten_batch=True,  # For variable-length sequence support\n            quant_config=quant_config,\n            prefix=prefix,\n        )\n\n    def forward(","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/siglip2.py#L169-L205","documentation":"Siglip2VisionAttention computes head_dim = hidden_size // num_attention_heads and verifies hidden_size is exactly divisible by num_heads. If not, attention projection dimensions would be inconsistent, so init fails immediately with the offending values.","triggerScenarios":"Loading a siglip2 vision config where config.hidden_size % config.num_attention_heads != 0 (e.g. an edited or corrupted config.json, or a checkpoint with non-standard dims).","commonSituations":"Hand-edited config.json during model conversion/merging, quantization tooling rewriting vision configs incorrectly, or using a mismatched vision config with a language model checkpoint.","solutions":["Restore the original config.json from the HF checkpoint (hidden_size/num_attention_heads of the vision tower)","If custom dims are intentional, choose num_attention_heads that divides hidden_size exactly","Verify you pointed --model-path at the complete, unmodified repo"],"exampleFix":"// config.json (vision)\n// before: {\"hidden_size\": 1152, \"num_attention_heads\": 10}\n// after: {\"hidden_size\": 1152, \"num_attention_heads\": 9}","handlingStrategy":"validation","validationCode":"assert config.hidden_size % config.num_attention_heads == 0, \\\n    (config.hidden_size, config.num_attention_heads)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate vision config divisibility right after loading config.json","Never hand-edit hidden_size/num_attention_heads independently"],"tags":["siglip2","vision-config","divisibility"],"backgroundTag":"config-divisibility-check-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}