hiyouga/LlamaFactory · error · ValueError

MOSS-VL video frame tokens do not match the processed video

Error message

MOSS-VL video frame tokens do not match the processed video after tokenization: tokens={video_frame_counts}, frames={expected_video_frames}. Please increase `cutoff_len` or reduce `video_maxlen`.

What it means

MOSS-VL post-tokenization check: the number of image tokens counted inside each reconstructed video block differs from expected_video_frames (the frame counts of the processed videos). Truncation inside a video block, or a frame-count mismatch between preprocessing and tokenization, produces different frame token counts.

Source

Thrown at src/llamafactory/data/mm_plugin.py:609

                    current_video_frames += 1
                else:
                    media_order.append("image")

        if in_video:
            raise ValueError(
                "MOSS-VL encountered an incomplete video token block after tokenization. "
                "Please increase `cutoff_len` or reduce `video_maxlen`."
            )

        if media_order.count("image") != num_images or media_order.count("video") != num_videos:
            raise ValueError(
                "MOSS-VL media tokens do not match the provided media after tokenization: "
                f"order={media_order}, images={num_images}, videos={num_videos}. "
                "Please increase `cutoff_len` if a visual placeholder was truncated."
            )

        if expected_video_frames is not None and video_frame_counts != expected_video_frames:
            raise ValueError(
                "MOSS-VL video frame tokens do not match the processed video after tokenization: "
                f"tokens={video_frame_counts}, frames={expected_video_frames}. "
                "Please increase `cutoff_len` or reduce `video_maxlen`."
            )

        return media_order

    @override
    def process_messages(
        self,
        messages: list[dict[str, str]],
        images: list["ImageInput"],
        videos: list["VideoInput"],
        audios: list["AudioInput"],
        processor: Optional["MMProcessor"],
    ) -> list[dict[str, str]]:
        self._validate_input(processor, images, videos, audios)
        self._validate_messages(messages, images, videos, audios)

View on GitHub (pinned to f28afaf635)

Solutions

  1. Increase cutoff_len or reduce video_maxlen so every frame token of every video fits.
  2. Pass consistent video_fps/video_maxlen kwargs to get_mm_plugin so regularization and tokenization agree.
  3. Check for duplicate <video> tags mapping to fewer actual videos (count mismatch upstream).

Example fix

### before
plugin = get_mm_plugin(name='moss_vl', video_maxlen=256)
cutoff_len: 4096
### after
plugin = get_mm_plugin(name='moss_vl', video_maxlen=64)
cutoff_len: 16384
Defensive patterns

Strategy: validation

Validate before calling

# verify each video block contains the expected frame token count
in_video, count, counts = False, 0, []
for t in ids:
    if t == processor.vision_start_token_id: in_video, count = True, 0
    elif t == processor.vision_end_token_id: counts.append(count); in_video = False
    elif t == processor.image_token_id and in_video: count += 1
assert counts == expected_video_frames, 'frame tokens truncated; raise cutoff_len or reduce video_maxlen'

Prevention

When it happens

Trigger: video placeholder truncated partway (some frame tokens cut), or the processor resamples videos differently than the plugin expected — e.g. video_maxlen / fps applied inconsistently between the regularize step and the tokenizer expansion.

Common situations: cutoff_len just barely fits a video so a few frame tokens get cut; mixing plugin kwargs (video_fps, video_maxlen) with processor defaults that disagree.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f2c23b041be46ecf. Report an issue: GitHub.