{"record":{"id":"31e4abae2dac9f42","repo":"hiyouga/LlamaFactory","slug":"moss-vl-encountered-an-incomplete-video-token-bloc","errorCode":null,"errorMessage":"MOSS-VL encountered an incomplete video token block after tokenization. Please increase `cutoff_len` or reduce `video_maxlen`.","messagePattern":"MOSS-VL encountered an incomplete video token block after tokenization\\. Please increase `cutoff_len` or reduce `video_maxlen`\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/data/mm_plugin.py","lineNumber":596,"sourceCode":"                in_video = True\n                current_video_frames = 0\n            elif token_id == processor.vision_end_token_id:\n                if not in_video:\n                    raise ValueError(\n                        \"MOSS-VL encountered a video end token without a matching start token after tokenization. \"\n                        \"Please increase `cutoff_len` if a video placeholder was truncated.\"\n                    )\n\n                video_frame_counts.append(current_video_frames)\n                in_video = False\n            elif token_id == processor.image_token_id:\n                if in_video:\n                    current_video_frames += 1\n                else:\n                    media_order.append(\"image\")\n\n        if in_video:\n            raise ValueError(\n                \"MOSS-VL encountered an incomplete video token block after tokenization. \"\n                \"Please increase `cutoff_len` or reduce `video_maxlen`.\"\n            )\n\n        if media_order.count(\"image\") != num_images or media_order.count(\"video\") != num_videos:\n            raise ValueError(\n                \"MOSS-VL media tokens do not match the provided media after tokenization: \"\n                f\"order={media_order}, images={num_images}, videos={num_videos}. \"\n                \"Please increase `cutoff_len` if a visual placeholder was truncated.\"\n            )\n\n        if expected_video_frames is not None and video_frame_counts != expected_video_frames:\n            raise ValueError(\n                \"MOSS-VL video frame tokens do not match the processed video after tokenization: \"\n                f\"tokens={video_frame_counts}, frames={expected_video_frames}. \"\n                \"Please increase `cutoff_len` or reduce `video_maxlen`.\"\n            )\n","sourceCodeStart":578,"sourceCodeEnd":614,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/mm_plugin.py#L578-L614","documentation":"MOSS-VL post-tokenization check: the token stream ends while still inside a video block (a vision_start was seen with no matching vision_end). The video placeholder was truncated before its terminating token, so frame counting cannot complete.","triggerScenarios":"A MOSS-VL sample whose expanded video placeholder exceeds the remaining cutoff_len budget; the tail of the sequence (containing vision_eos) is dropped. The error message itself points at cutoff_len or video_maxlen.","commonSituations":"High-resolution / many-frame videos; cutoff_len tuned for text-only data; video_maxlen default too generous for the chosen cutoff.","solutions":["Increase cutoff_len to cover the largest video placeholder in the dataset.","Reduce video_maxlen / video_fps so each video yields fewer frame tokens.","Pre-compute worst-case placeholder length (frames x tokens-per-frame) and set cutoff_len above it."],"exampleFix":"### before\ncutoff_len: 4096\n### after\ncutoff_len: 32768\nvideo_maxlen: 128","handlingStrategy":"validation","validationCode":"ids = tokenizer(sample_text_with_placeholders).input_ids\nopen_blocks = 0\nfor t in ids:\n    if t == processor.vision_start_token_id: open_blocks += 1\n    elif t == processor.vision_end_token_id: open_blocks -= 1\nassert open_blocks == 0, 'truncated video placeholder; increase cutoff_len / reduce video_maxlen'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Budget cutoff_len as max_text_len + total_media_tokens with margin.","Cap videos per sample when cutoff_len is fixed."],"tags":["multimodal","moss-vl","video","truncation","cutoff-len","video-maxlen"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}