sgl-project/sglang · error · ValueError

embed_dim must be divisible by num_heads (got `embed_dim`: {

Error message

embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads}).

What it means

Siglip2VisionAttention computes head_dim = hidden_size // num_attention_heads and verifies hidden_size is exactly divisible by num_heads. If not, attention projection dimensions would be inconsistent, so init fails immediately with the offending values.

Source

Thrown at python/sglang/srt/models/siglip2.py:187

class Siglip2Attention(nn.Module):
    """Multi-headed attention for Siglip2 using optimized VisionAttention backend."""

    def __init__(
        self,
        config: Siglip2VisionConfig,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ):
        super().__init__()
        self.config = config
        self.embed_dim = config.hidden_size
        self.num_heads = config.num_attention_heads
        self.head_dim = self.embed_dim // self.num_heads

        if self.head_dim * self.num_heads != self.embed_dim:
            raise ValueError(
                f"embed_dim must be divisible by num_heads "
                f"(got `embed_dim`: {self.embed_dim} and `num_heads`:"
                f" {self.num_heads})."
            )

        # Use SGLang's optimized VisionAttention with automatic backend selection
        self.attn = VisionAttention(
            embed_dim=self.embed_dim,
            num_heads=self.num_heads,
            projection_size=self.embed_dim,
            use_qkv_parallel=True,
            dropout=config.attention_dropout,
            flatten_batch=True,  # For variable-length sequence support
            quant_config=quant_config,
            prefix=prefix,
        )

    def forward(

View on GitHub (pinned to 0132848349)

Solutions

1. Restore the original config.json from the HF checkpoint (hidden_size/num_attention_heads of the vision tower)
2. If custom dims are intentional, choose num_attention_heads that divides hidden_size exactly
3. Verify you pointed --model-path at the complete, unmodified repo

Example fix

// config.json (vision)
// before: {"hidden_size": 1152, "num_attention_heads": 10}
// after: {"hidden_size": 1152, "num_attention_heads": 9}

assert config.hidden_size % config.num_attention_heads == 0, \
    (config.hidden_size, config.num_attention_heads)

When it happens

Trigger: Loading a siglip2 vision config where config.hidden_size % config.num_attention_heads != 0 (e.g. an edited or corrupted config.json, or a checkpoint with non-standard dims).

Common situations: Hand-edited config.json during model conversion/merging, quantization tooling rewriting vision configs incorrectly, or using a mismatched vision config with a language model checkpoint.

Related errors

• world_size must be positive and divide global_heads
• Invalid spatial patching for packed token latents. Expected
• Cosmos3 action_horizon must be divisible by 4 so num_frames
• raw_latent_shape must be divisible by patch_size for SAP att
• dim {dim} must be divisible by head_dim {head_dim}.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f1188eb8e2cdaef1. Report an issue: GitHub.