{"record":{"id":"27b00b7cf7486c7b","repo":"hiyouga/LlamaFactory","slug":"each-video-placeholder-must-be-followed-by-an-a","errorCode":null,"errorMessage":"Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACEHOLDER} when using audio in video.","messagePattern":"Each (.+?) must be followed by an (.+?) when using audio in video\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/data/mm_plugin.py","lineNumber":3028,"sourceCode":"                    IMAGE_PLACEHOLDER,\n                    f\"{self.vision_bos_token}{self.image_token * image_seqlen}{self.vision_eos_token}\",\n                    1,\n                )\n                num_image_tokens += 1\n\n            if (\n                use_audio_in_video and len(audios) and len(videos)\n            ):  # if use the audio of video # deal video token and audio token together\n                if len(videos) != len(audios):\n                    raise ValueError(\n                        f\"Number of videos ({len(videos)}) must match number of audios ({len(audios)}) when using audio in video.\"\n                    )\n\n                while VIDEO_PLACEHOLDER in content:\n                    video_pos = content.find(VIDEO_PLACEHOLDER)\n                    audio_pos = content.find(AUDIO_PLACEHOLDER, video_pos)\n                    if audio_pos == -1 or audio_pos < video_pos:\n                        raise ValueError(\n                            f\"Each {VIDEO_PLACEHOLDER} must be followed by an {AUDIO_PLACEHOLDER} when using audio in video.\"\n                        )\n\n                    position_id_per_seconds: int = getattr(processor, \"position_id_per_seconds\", 25)\n                    audio_t_index = torch.arange(audio_lengths[num_audio_tokens])\n                    video_t_index = (\n                        torch.arange(video_grid_thw[num_video_tokens][0])\n                        .view(-1, 1, 1)\n                        .expand(\n                            -1,\n                            video_grid_thw[num_video_tokens][1] // image_processor.merge_size,\n                            video_grid_thw[num_video_tokens][2] // image_processor.merge_size,\n                        )\n                        .flatten()\n                        * mm_inputs[\"video_second_per_grid\"][num_video_tokens]\n                        * position_id_per_seconds\n                    ).long()\n                    t_ntoken_per_chunk = position_id_per_seconds * 2","sourceCodeStart":3010,"sourceCodeEnd":3046,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/mm_plugin.py#L3010-L3046","documentation":"With use_audio_in_video enabled, the Qwen2-Omni plugin requires each <video> placeholder in the message content to be followed (later in the string) by an <audio> placeholder, because it substitutes the video and its audio track as one fused token block with aligned timestamps.","triggerScenarios":"A sample whose content contains <video> but the next <audio> placeholder is missing entirely (content.find returns -1) — e.g. only <video> in the prompt while audios list is non-empty, or audio placeholder placed before the video.","commonSituations":"Reordering placeholders in the prompt template; writing '<audio> then <video>' instead of '<video> then <audio>'; forgetting the <audio> tag when adding an audio column.","solutions":["Rewrite the content so every <video> is immediately followed by its <audio>: e.g. 'Describe <video><audio>'.","Ensure placeholder ordering is video-then-audio in every turn that uses audio-in-video.","If audio is separate from the video (not the video's soundtrack), disable use_audio_in_video."],"exampleFix":"// before\n\"content\": \"What is said in <audio> and shown in <video>?\"\n// after\n\"content\": \"What is shown and said in <video><audio>?\"","handlingStrategy":"validation","validationCode":"def audio_in_video_ok(content):\n    v = content.find('<video>')\n    while v != -1:\n        a = content.find('<audio>', v)\n        if a == -1:\n            return False\n        v = content.find('<video>', a)\n    return True\nassert audio_in_video_ok(content)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize the fused placeholder pattern '<video><audio>' in every audio-in-video prompt.","Never reorder placeholders manually; build prompts from a template function."],"tags":["multimodal","qwen2-omni","audio","video","placeholder-order"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}