hiyouga/LlamaFactory · error · ValueError

Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACE

Error message

Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACEHOLDER} when using audio in video.

What it means

With use_audio_in_video enabled, the Qwen2-Omni plugin requires each <video> placeholder in the message content to be followed (later in the string) by an <audio> placeholder, because it substitutes the video and its audio track as one fused token block with aligned timestamps.

Source

Thrown at src/llamafactory/data/mm_plugin.py:3028

                    IMAGE_PLACEHOLDER,
                    f"{self.vision_bos_token}{self.image_token * image_seqlen}{self.vision_eos_token}",
                    1,
                )
                num_image_tokens += 1

            if (
                use_audio_in_video and len(audios) and len(videos)
            ):  # if use the audio of video # deal video token and audio token together
                if len(videos) != len(audios):
                    raise ValueError(
                        f"Number of videos ({len(videos)}) must match number of audios ({len(audios)}) when using audio in video."
                    )

                while VIDEO_PLACEHOLDER in content:
                    video_pos = content.find(VIDEO_PLACEHOLDER)
                    audio_pos = content.find(AUDIO_PLACEHOLDER, video_pos)
                    if audio_pos == -1 or audio_pos < video_pos:
                        raise ValueError(
                            f"Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACEHOLDER} when using audio in video."
                        )

                    position_id_per_seconds: int = getattr(processor, "position_id_per_seconds", 25)
                    audio_t_index = torch.arange(audio_lengths[num_audio_tokens])
                    video_t_index = (
                        torch.arange(video_grid_thw[num_video_tokens][0])
                        .view(-1, 1, 1)
                        .expand(
                            -1,
                            video_grid_thw[num_video_tokens][1] // image_processor.merge_size,
                            video_grid_thw[num_video_tokens][2] // image_processor.merge_size,
                        )
                        .flatten()
                        * mm_inputs["video_second_per_grid"][num_video_tokens]
                        * position_id_per_seconds
                    ).long()
                    t_ntoken_per_chunk = position_id_per_seconds * 2

View on GitHub (pinned to f28afaf635)

Solutions

  1. Rewrite the content so every <video> is immediately followed by its <audio>: e.g. 'Describe <video><audio>'.
  2. Ensure placeholder ordering is video-then-audio in every turn that uses audio-in-video.
  3. If audio is separate from the video (not the video's soundtrack), disable use_audio_in_video.

Example fix

// before
"content": "What is said in <audio> and shown in <video>?"
// after
"content": "What is shown and said in <video><audio>?"
Defensive patterns

Strategy: validation

Validate before calling

def audio_in_video_ok(content):
    v = content.find('<video>')
    while v != -1:
        a = content.find('<audio>', v)
        if a == -1:
            return False
        v = content.find('<video>', a)
    return True
assert audio_in_video_ok(content)

Prevention

When it happens

Trigger: A sample whose content contains <video> but the next <audio> placeholder is missing entirely (content.find returns -1) — e.g. only <video> in the prompt while audios list is non-empty, or audio placeholder placed before the video.

Common situations: Reordering placeholders in the prompt template; writing '<audio> then <video>' instead of '<video> then <audio>'; forgetting the <audio> tag when adding an audio column.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/27b00b7cf7486c7b. Report an issue: GitHub.