What causes "unexpected shape for qkv {:?}" in huggingface/candle?

Verify hidden_size and num_heads in Config match the checkpoint so the qkv projection yields dim 2 == 3

unexpected shape for qkv

Error message

unexpected shape for qkv {:?}

What it means

apply_rotary_emb_qkv expects qkv shaped (b, seqlen, 3, heads, head_dim) where dim 2 is exactly 3 (q, k, v). When the third dimension is not 3, the tensor layout is not the packed QKV layout the MixFormer code expects, so it bails with the offending shape.

Solutions

Verify hidden_size and num_heads in Config match the checkpoint so the qkv projection yields dim 2 == 3
Do not pre-split or reshape the qkv tensor before apply_rotary_emb_qkv
Inspect the printed shape in the error and fix the reshape/projection that produced it

Example fix

// before
let qkv = qkv.reshape((b, seq, heads * head_dim * 2, head_dim))?; // dim2 != 3
// after
let qkv = qkv.reshape((b, seq, 3, heads, head_dim))?; // packed q,k,v

Defensive patterns

Strategy: validation

Validate before calling

let (b, seq, three, heads, hd) = qkv.dims5()?;
if three != 3 {
    return Err(anyhow::anyhow!("qkv dim 2 must be 3, got {three}"));
}

Type guard

fn is_packed_qkv(qkv: &Tensor) -> candle::Result<bool> {
    Ok(qkv.dims5()?.2 == 3)
}

Try / catch

let (q, k, v) = match apply_rotary_emb_qkv(&qkv, offset) {
    Ok(v) => v,
    Err(e) if e.to_string().contains("unexpected shape for qkv") => {
        return Err(anyhow::anyhow!("check hidden_size/num_heads; qkv must be (b, seq, 3, heads, hd)"))
    }
    Err(e) => return Err(e.into()),
};

Prevention

Assert hidden_size == num_heads * head_dim in the config
Never reshape or pre-split the packed qkv tensor before rotary embedding
Print tensor shapes when wiring new model variants into MixFormer

When it happens

Trigger: Calling MixFormer forward paths (via apply_rotary_emb_qkv) with a qkv tensor whose dim 2 != 3, usually caused by a wrong head-count/hidden-size config producing a mis-shaped projection, or feeding a pre-split q/k/v tensor into the packed path.

Common situations: Config mismatch between hidden_size/num_heads and the checkpoint weights; manually reshaping or splitting qkv before calling; using a model variant that packs attention differently.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/3c8c15bc736e13b3. Report an issue: GitHub.

Appendix: source

Thrown at candle-transformers/src/models/mixformer.rs:174

        let inv_freq = Tensor::from_vec(inv_freq, (1, inv_freq_len), dev)?;
        let t = Tensor::arange(0u32, max_seq_len as u32, dev)?
            .to_dtype(DType::F32)?
            .reshape((max_seq_len, 1))?;
        let freqs = t.matmul(&inv_freq)?;
        Ok(Self {
            sin: freqs.sin()?.to_dtype(dtype)?,
            cos: freqs.cos()?.to_dtype(dtype)?,
        })
    }

    fn apply_rotary_emb_qkv(
        &self,
        qkv: &Tensor,
        seqlen_offset: usize,
    ) -> Result<(Tensor, Tensor, Tensor)> {
        let (_b_size, seqlen, three, _, _headdim) = qkv.dims5()?;
        if three != 3 {
            candle::bail!("unexpected shape for qkv {:?}", qkv.shape())
        }
        let (_rotary_seqlen, rotary_dim) = self.cos.dims2()?;
        let rotary_dim = rotary_dim * 2;
        let q_rot = qkv.i((.., .., 0, .., ..rotary_dim))?.contiguous()?;
        let q_pass = qkv.i((.., .., 0, .., rotary_dim..))?;
        let k_rot = qkv.i((.., .., 1, .., ..rotary_dim))?.contiguous()?;
        let k_pass = qkv.i((.., .., 1, .., rotary_dim..))?;
        let c = self.cos.narrow(0, seqlen_offset, seqlen)?;
        let s = self.sin.narrow(0, seqlen_offset, seqlen)?;
        let q_rot = candle_nn::rotary_emb::rope_thd(&q_rot, &c, &s)?;
        let k_rot = candle_nn::rotary_emb::rope_thd(&k_rot, &c, &s)?;
        let q = Tensor::cat(&[&q_rot, &q_pass], D::Minus1)?;
        let k = Tensor::cat(&[&k_rot, &k_pass], D::Minus1)?;
        let v = qkv.i((.., .., 2))?;
        Ok((q, k, v))
    }
}

View on GitHub (pinned to d5fee525bf)