What causes "embed_dim must be divisible by num_heads ({} % {} != 0)" in huggingface/candle?

Set num_attention_heads to a divisor of hidden_size in the config (e.g. 16 heads for hidden_size 1024, head_dim 64)

embed_dim must be divisible by num_heads ({} % {} != 0)

Error message

embed_dim must be divisible by num_heads ({} % {} != 0)

What it means

VoxtralEncoderLayer (attention construction) computes head_dim = embed_dim / num_heads and then verifies head_dim * num_heads == embed_dim. Integer division would otherwise silently drop dimensions, so a non-divisible configuration bails with the embed_dim % num_heads mismatch. This is a config sanity check at encoder construction time.

Source

Thrown at candle-transformers/src/models/voxtral/model.rs:278

struct VoxtralAttention {
    q_proj: Linear,
    k_proj: Linear,
    v_proj: Linear,
    out_proj: Linear,
    num_heads: usize,
    head_dim: usize,
    scaling: f64,
    attention_dropout: Dropout,
}

impl VoxtralAttention {
    fn new(cfg: &VoxtralEncoderConfig, vb: VarBuilder) -> Result<Self> {
        let embed_dim = cfg.hidden_size;
        let num_heads = cfg.num_attention_heads;
        let head_dim = embed_dim / num_heads;

        if head_dim * num_heads != embed_dim {
            candle::bail!(
                "embed_dim must be divisible by num_heads ({} % {} != 0)",
                embed_dim,
                num_heads
            );
        }

        let scaling = (head_dim as f64).powf(-0.5);

        let q_proj = linear(embed_dim, embed_dim, vb.pp("q_proj"))?;
        let k_proj = linear_no_bias(embed_dim, embed_dim, vb.pp("k_proj"))?;
        let v_proj = linear(embed_dim, embed_dim, vb.pp("v_proj"))?;
        let out_proj = linear(embed_dim, embed_dim, vb.pp("out_proj"))?;

        let attention_dropout = Dropout::new(cfg.attention_dropout as f32);

        Ok(Self {
            q_proj,
            k_proj,

View on GitHub (pinned to d5fee525bf)

Solutions

Set num_attention_heads to a divisor of hidden_size in the config (e.g. 16 heads for hidden_size 1024, head_dim 64)
Use the official checkpoint's config values verbatim instead of hand-editing
Compute heads from a desired head_dim: heads = hidden_size / head_dim

Example fix

// before
let cfg = VoxtralEncoderConfig { hidden_size: 1024, num_attention_heads: 30, .. };
// after
let cfg = VoxtralEncoderConfig { hidden_size: 1024, num_attention_heads: 16, .. };

Defensive patterns

Strategy: validation

Validate before calling

if cfg.hidden_size % cfg.num_attention_heads != 0 {
    return Err(anyhow::anyhow!("hidden_size {} not divisible by heads {}", cfg.hidden_size, cfg.num_attention_heads));
}

Try / catch

match VoxtralEncoder::new(&cfg, vb) {
    Err(e) if e.to_string().contains("divisible by num_heads") => {
        anyhow::bail!("fix VoxtralEncoderConfig head/embed_dim values")
    }
    r => r?,
}

Prevention

Copy hidden_size/num_attention_heads verbatim from the checkpoint config
Prefer standard head_dim values (64/80/128) when editing configs
Add a config validation helper run before model construction

When it happens

Trigger: Calling VoxtralEncoderLayer::new (during VoxtralEncoder/model construction) with a VoxtralEncoderConfig where hidden_size is not an exact multiple of num_attention_heads.

Common situations: Hand-written or edited VoxtralEncoderConfig values; porting a config where hidden_size was changed (e.g. distilled model) without adjusting head count; typo like hidden_size 1024 with 30 heads.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/6086e7b6db7775a9. Report an issue: GitHub.