{"record":{"id":"6086e7b6db7775a9","repo":"huggingface/candle","slug":"embed-dim-must-be-divisible-by-num-heads","errorCode":null,"errorMessage":"embed_dim must be divisible by num_heads ({} % {} != 0)","messagePattern":"embed_dim must be divisible by num_heads \\((.+?) % (.+?) != 0\\)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-transformers/src/models/voxtral/model.rs","lineNumber":278,"sourceCode":"struct VoxtralAttention {\n    q_proj: Linear,\n    k_proj: Linear,\n    v_proj: Linear,\n    out_proj: Linear,\n    num_heads: usize,\n    head_dim: usize,\n    scaling: f64,\n    attention_dropout: Dropout,\n}\n\nimpl VoxtralAttention {\n    fn new(cfg: &VoxtralEncoderConfig, vb: VarBuilder) -> Result<Self> {\n        let embed_dim = cfg.hidden_size;\n        let num_heads = cfg.num_attention_heads;\n        let head_dim = embed_dim / num_heads;\n\n        if head_dim * num_heads != embed_dim {\n            candle::bail!(\n                \"embed_dim must be divisible by num_heads ({} % {} != 0)\",\n                embed_dim,\n                num_heads\n            );\n        }\n\n        let scaling = (head_dim as f64).powf(-0.5);\n\n        let q_proj = linear(embed_dim, embed_dim, vb.pp(\"q_proj\"))?;\n        let k_proj = linear_no_bias(embed_dim, embed_dim, vb.pp(\"k_proj\"))?;\n        let v_proj = linear(embed_dim, embed_dim, vb.pp(\"v_proj\"))?;\n        let out_proj = linear(embed_dim, embed_dim, vb.pp(\"out_proj\"))?;\n\n        let attention_dropout = Dropout::new(cfg.attention_dropout as f32);\n\n        Ok(Self {\n            q_proj,\n            k_proj,","sourceCodeStart":260,"sourceCodeEnd":296,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-transformers/src/models/voxtral/model.rs#L260-L296","documentation":"VoxtralEncoderLayer (attention construction) computes head_dim = embed_dim / num_heads and then verifies head_dim * num_heads == embed_dim. Integer division would otherwise silently drop dimensions, so a non-divisible configuration bails with the embed_dim % num_heads mismatch. This is a config sanity check at encoder construction time.","triggerScenarios":"Calling VoxtralEncoderLayer::new (during VoxtralEncoder/model construction) with a VoxtralEncoderConfig where hidden_size is not an exact multiple of num_attention_heads.","commonSituations":"Hand-written or edited VoxtralEncoderConfig values; porting a config where hidden_size was changed (e.g. distilled model) without adjusting head count; typo like hidden_size 1024 with 30 heads.","solutions":["Set num_attention_heads to a divisor of hidden_size in the config (e.g. 16 heads for hidden_size 1024, head_dim 64)","Use the official checkpoint's config values verbatim instead of hand-editing","Compute heads from a desired head_dim: heads = hidden_size / head_dim"],"exampleFix":"// before\nlet cfg = VoxtralEncoderConfig { hidden_size: 1024, num_attention_heads: 30, .. };\n// after\nlet cfg = VoxtralEncoderConfig { hidden_size: 1024, num_attention_heads: 16, .. };","handlingStrategy":"validation","validationCode":"if cfg.hidden_size % cfg.num_attention_heads != 0 {\n    return Err(anyhow::anyhow!(\"hidden_size {} not divisible by heads {}\", cfg.hidden_size, cfg.num_attention_heads));\n}","typeGuard":null,"tryCatchPattern":"match VoxtralEncoder::new(&cfg, vb) {\n    Err(e) if e.to_string().contains(\"divisible by num_heads\") => {\n        anyhow::bail!(\"fix VoxtralEncoderConfig head/embed_dim values\")\n    }\n    r => r?,\n}","preventionTips":["Copy hidden_size/num_attention_heads verbatim from the checkpoint config","Prefer standard head_dim values (64/80/128) when editing configs","Add a config validation helper run before model construction"],"tags":["rust","candle","voxtral","config","shape-validation"],"backgroundTag":"embed-dim-not-divisible-by-heads","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}