hiyouga/LlamaFactory · error · ValueError

Number of videos ({len(videos)}) must match number of audios

Error message

Number of videos ({len(videos)}) must match number of audios ({len(audios)}) when using audio in video.

What it means

In the Qwen2-Omni plugin, when use_audio_in_video is enabled and both audios and videos are present, every video must carry its own audio track: len(videos) must equal len(audios). The plugin then fuses each video's visual tokens with the corresponding audio's tokens using a shared time index.

Source

Thrown at src/llamafactory/data/mm_plugin.py:3020

            video_grid_thw = [None] * len(videos)
            audio_lengths = [None] * len(audios)

        for message in messages:
            content = message["content"]
            while IMAGE_PLACEHOLDER in content:
                image_seqlen = image_grid_thw[num_image_tokens].prod() // merge_length if self.expand_mm_tokens else 1
                content = content.replace(
                    IMAGE_PLACEHOLDER,
                    f"{self.vision_bos_token}{self.image_token * image_seqlen}{self.vision_eos_token}",
                    1,
                )
                num_image_tokens += 1

            if (
                use_audio_in_video and len(audios) and len(videos)
            ):  # if use the audio of video # deal video token and audio token together
                if len(videos) != len(audios):
                    raise ValueError(
                        f"Number of videos ({len(videos)}) must match number of audios ({len(audios)}) when using audio in video."
                    )

                while VIDEO_PLACEHOLDER in content:
                    video_pos = content.find(VIDEO_PLACEHOLDER)
                    audio_pos = content.find(AUDIO_PLACEHOLDER, video_pos)
                    if audio_pos == -1 or audio_pos < video_pos:
                        raise ValueError(
                            f"Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACEHOLDER} when using audio in video."
                        )

                    position_id_per_seconds: int = getattr(processor, "position_id_per_seconds", 25)
                    audio_t_index = torch.arange(audio_lengths[num_audio_tokens])
                    video_t_index = (
                        torch.arange(video_grid_thw[num_video_tokens][0])
                        .view(-1, 1, 1)
                        .expand(
                            -1,

View on GitHub (pinned to f28afaf635)

Solutions

1. Make the audios list length equal the videos list length — supply one audio track per video (even silent placeholders if the model expects them).
2. If videos have no meaningful audio, disable use_audio_in_video in the template config.
3. Filter rows where the counts differ before training.

Example fix

### before
{"content": "<video>", "videos": ["a.mp4", "b.mp4"], "audios": ["a.wav"]}
### after
{"content": "<video>", "videos": ["a.mp4", "b.mp4"], "audios": ["a.wav", "b.wav"]}

if use_audio_in_video and audios and videos:
    assert len(videos) == len(audios), f'{len(videos)} videos vs {len(audios)} audios'

When it happens

Trigger: Template config with use_audio_in_video=true plus a dataset where a sample provides 2 <video> placeholders but 1 audio (or vice versa), including videos with no accompanying audio files.

Common situations: Enabling audio-in-video on a video-only dataset (audios column present for some rows only); misaligned video/audio columns after data conversion.

Related errors

• Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACE
• The number of videos does not match the number of {VIDEO_PLA
• This model does not support video input. Please check whethe
• This model does not support audio input. Please check whethe
• Audio feature extractor was not found, please check and upda

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a2ce0e64030b2c1a. Report an issue: GitHub.