Video embedding length {} does not match placeholder tokens

Error message

Video embedding length {} does not match placeholder tokens {}

What it means

Thrown by qwen3_vl's forward when the video embedding row count from the vision tower differs from the total number of video placeholder token positions (continuous_vid_pad span lengths) in input_ids. Every video placeholder must map to exactly one video embedding row.

Source

Thrown at candle-transformers/src/models/qwen3_vl/mod.rs:165

                pixel_values = pixel_values.reshape((dims[0] * dims[1], dims[2]))?;
            }
            let (video_embeds, deepstack_video_embeds) =
                self.vision.forward(&pixel_values, video_grid_thw_ref)?;
            let video_embeds = video_embeds.to_device(&device)?.to_dtype(self.text.dtype)?;
            let mut deepstack_video_embeds = deepstack_video_embeds
                .into_iter()
                .map(|t| t.to_device(&device)?.to_dtype(self.text.dtype))
                .collect::<Result<Vec<_>>>()?;

            let mut offset = 0usize;
            let mut video_mask =
                Tensor::zeros((batch_size, seq_len), DType::F32, input_ids.device())?;
            let total_expected: usize = continuous_vid_pad
                .iter()
                .flat_map(|spans| spans.iter().map(|(s, e)| e - s))
                .sum();
            if video_embeds.dim(0)? != total_expected {
                candle::bail!(
                    "Video embedding length {} does not match placeholder tokens {}",
                    video_embeds.dim(0)?,
                    total_expected
                );
            }

            for (batch, spans) in continuous_vid_pad.iter().enumerate() {
                for &(start, end) in spans {
                    let len = end - start;
                    let chunk = video_embeds.narrow(0, offset, len)?;
                    offset += len;
                    input_embeds = input_embeds.slice_assign(
                        &[batch..batch + 1, start..end, 0..hidden_dim],
                        &chunk.unsqueeze(0)?,
                    )?;
                    let ones = Tensor::ones((1, len), DType::F32, input_ids.device())?;
                    video_mask = video_mask.slice_assign(&[batch..batch + 1, start..end], &ones)?;
                }

View on GitHub (pinned to d5fee525bf)

Solutions

Generate input_ids, pixel_values_videos and video_grid_thw in one processor call so frame/patch counts and placeholder counts agree
Verify video_grid_thw's temporal dimension matches the number of sampled frames encoded in the placeholders
Ensure pixel_values_videos and input_ids come from the same example/batch

Example fix

// before (frames re-sampled after tokenization)
let input_ids = tokenize(text_with_16_frames);
let (video_pixels, video_grid_thw) = encode_video(video, /*frames=*/32)?;
model.forward(&input_ids, None, None, Some(&video_pixels), Some(&video_grid_thw))?;
// after
let inputs = processor(text, video)?; // consistent placeholder & patch counts
model.forward(&inputs.input_ids, None, None, Some(&inputs.pixel_values_videos), Some(&inputs.video_grid_thw))?;

Defensive patterns

Strategy: validation

Validate before calling

let total_vid_pads: usize = vid_spans.iter().flat_map(|s| s.iter().map(|(a,b)| b-a)).sum();
if video_embeds.dim(0)? != total_vid_pads {
    anyhow::bail!("video patch count {} != placeholder count {}", video_embeds.dim(0)?, total_vid_pads);
}

Try / catch

match model.forward(&input_ids, None, None, Some(&video_pixels), Some(&video_grid_thw)) {
    Ok(l) => l,
    Err(e) if e.to_string().contains("does not match placeholder tokens") => {
        anyhow::bail!("video frames/grid out of sync with tokenization; re-run processor")
    }
    Err(e) => return Err(e.into()),
}

Prevention

Fix frame sampling before tokenization; never re-sample after
Verify video_grid_thw temporal dim equals sampled frame count
Group all video inputs with their tokenization in one processor pass

When it happens

Trigger: Calling forward where vision_encoder output rows != sum of (end-start) across continuous video placeholder spans — wrong video_grid_thw, temporal/patch preprocessing mismatch, or input_ids containing a stale number of <|video_pad|> tokens.

Common situations: Frame sampling count differing between tokenization time and pixel-value generation; grid_thw temporal dim not matching sampled frames; mixing videos between examples in a batch; processor version drift.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/22e030cfdc27b967. Report an issue: GitHub.