What causes "only multi_query_attention=true is supported" in huggingface/candle?

Set "multi_query_attention": true in the model config before loading

only multi_query_attention=true is supported

Error message

only multi_query_attention=true is supported

What it means

GLM-4 attention in candle only implements multi-query attention (MQA/GQA) where key/value heads are fewer than query heads. Attention::forward checks the config flag multi_query_attention and bails if the checkpoint uses standard multi-head attention, since the narrow-based QKV splitting assumes that layout.

Source

Thrown at candle-transformers/src/models/glm4.rs:286

            num_multi_query_groups_per_partition: cfg.multi_query_group_num,
            hidden_size_per_attention_head: cfg.kv_channels,
            kv_cache: None,
        })
    }

    fn reset_kv_cache(&mut self) {
        self.kv_cache = None
    }

    fn forward(
        &mut self,
        xs: &Tensor,
        attention_mask: &Option<Tensor>,
        rotary_emb: &RotaryEmbedding,
    ) -> Result<Tensor> {
        let mixed_x_layer = xs.apply(&self.query_key_value)?;
        if !self.multi_query_attention {
            candle::bail!("only multi_query_attention=true is supported")
        }
        let hpa = self.hidden_size_per_attention_head;
        let query_layer =
            mixed_x_layer.narrow(D::Minus1, 0, self.num_attention_heads_per_partition * hpa)?;
        let key_layer = mixed_x_layer.narrow(
            D::Minus1,
            self.num_attention_heads_per_partition * hpa,
            self.num_multi_query_groups_per_partition * hpa,
        )?;
        let value_layer = mixed_x_layer.narrow(
            D::Minus1,
            self.num_attention_heads_per_partition * hpa
                + self.num_multi_query_groups_per_partition * hpa,
            self.num_multi_query_groups_per_partition * hpa,
        )?;
        let query_layer = query_layer.reshape((
            query_layer.dim(0)?,
            query_layer.dim(1)?,

View on GitHub (pinned to d5fee525bf)

Solutions

Set "multi_query_attention": true in the model config before loading
Verify you downloaded the intended GLM-4 checkpoint variant that uses MQA/GQA
Extend the candle implementation if you genuinely need MHA support

Example fix

// before (config.json)
{"multi_query_attention": false, ...}
// after
{"multi_query_attention": true, ...}

Defensive patterns

Strategy: validation

Validate before calling

if cfg.multi_query_attention != true {
    return Err("this checkpoint is not multi-query attention; candle glm4 only supports MQA");
}

Type guard

fn is_mqa(c: &glm4::Config) -> bool { c.multi_query_attention }

Try / catch

match model.forward(&xs, &positions, &mask) {
    Err(e) if e.to_string().contains("only multi_query_attention") =>
        Err(anyhow!("set multi_query_attention=true or use a different model impl")),
    r => r.map_err(Into::into),
}

Prevention

Check multi_query_attention in config.json before loading GLM-4 checkpoints
Download the intended GLM-4 variant known to use MQA/GQA
Don't reuse configs from other GLM sizes without review

When it happens

Trigger: Loading a GLM-4 checkpoint whose config has multi_query_attention=false (or missing, defaulting to false) and running a forward pass.

Common situations: Using a base GLM variant/config that doesn't set multi_query_attention=true; a truncated or hand-made config.json missing the key; adapting code from another GLM size that used MHA.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/62417b2ec8f8890c. Report an issue: GitHub.