hiyouga/LlamaFactory · error · ValueError

MiniCPM-V model does not support input images and videos at

Error message

MiniCPM-V model does not support input images and videos at the same time.

What it means

The MiniCPM-V plugin's process_messages refuses samples that contain both images and videos: the MiniCPM-V architecture and its chat template cannot interleave the two modalities in one conversation.

Source

Thrown at src/llamafactory/data/mm_plugin.py:1648

        return mm_inputs

    @override
    def process_messages(
        self,
        messages: list[dict[str, str]],
        images: list["ImageInput"],
        videos: list["VideoInput"],
        audios: list["AudioInput"],
        processor: Optional["MMProcessor"],
    ) -> list[dict[str, str]]:
        self._validate_input(processor, images, videos, audios)
        self._validate_messages(messages, images, videos, audios)
        num_image_tokens, num_video_tokens, num_audio_tokens = 0, 0, 0
        messages = deepcopy(messages)
        image_processor: BaseImageProcessor = getattr(processor, "image_processor")
        mm_inputs, audio_inputs = {}, {}
        if len(images) != 0 and len(videos) != 0:
            raise ValueError("MiniCPM-V model does not support input images and videos at the same time.")

        if len(videos) != 0:
            max_slice_nums = 2
            use_image_id = False
            mm_inputs = self._get_mm_inputs([], videos, [], processor)
        else:
            max_slice_nums = image_processor.max_slice_nums
            use_image_id = image_processor.use_image_id

        for i, message in enumerate(messages):
            content = message["content"]
            while IMAGE_PLACEHOLDER in content:
                content = content.replace(IMAGE_PLACEHOLDER, "{{image}}", 1)
                num_image_tokens += 1

            while VIDEO_PLACEHOLDER in content:
                video_seqlen = len(mm_inputs["image_sizes"][num_video_tokens]) if self.expand_mm_tokens else 1
                content = content.replace(VIDEO_PLACEHOLDER, "{{image}}" * video_seqlen, 1)

View on GitHub (pinned to f28afaf635)

Solutions

  1. Split the sample: keep either the images or the videos, and remove the other modality's placeholders.
  2. Filter mixed rows from the dataset before training with MiniCPM-V.
  3. Use a model/plugin that supports mixed media (e.g. qwen2_vl/qwen3_vl) if mixed samples are required.

Example fix

// before
{"content": "<image> and <video>", "images": ["a.jpg"], "videos": ["b.mp4"]}
// after
{"content": "<video>", "videos": ["b.mp4"]}  // image removed for MiniCPM-V
Defensive patterns

Strategy: validation

Validate before calling

assert not (images and videos), 'MiniCPM-V cannot mix images and videos in one sample'

Prevention

When it happens

Trigger: Using template 'minicpm_v' with a dataset row whose messages contain both <img>/(image) and <video> placeholders, or whose images and videos columns are both non-empty.

Common situations: Mixed-media datasets reused across models; a row that always carries an images column (e.g. empty string still counts) alongside videos.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6eccd042d0283a92. Report an issue: GitHub.