hiyouga/LlamaFactory · error · RuntimeError

Processor did not emit {modality} placeholder tokens for the

Error message

Processor did not emit {modality} placeholder tokens for the dummy sample.

What it means

To build a dummy media fragment, the renderer renders a minimal image/video/audio conversation and expects the processor to emit mm_token_type_ids containing the modality's marker id (1=image, 2=video, 3=audio) plus a presence key like pixel_values/input_features. If the processor does not tag tokens that way (non-conforming or partially supported multimodal processor), fragment extraction cannot proceed and a RuntimeError is raised.

Source

Thrown at src/llamafactory/v1/core/rendering/rendering.py:243

        elif modality == "video":
            # A minimal clip: the temporal patch size is typically 2, so provide two frames.
            media_block = {"type": "video_url", "value": np.zeros((2, 64, 64, 3), dtype=np.uint8)}
            target, presence_key = 2, "pixel_values_videos"
        else:
            # A short synthetic waveform at the model's sampling rate; the feature extractor pads it.
            sr = self.processor.feature_extractor.sampling_rate
            media_block = {"type": "audio_url", "value": np.zeros(sr // 10, dtype=np.float32)}
            target, presence_key = 3, "input_features"

        messages: list[Message] = [
            {"role": "user", "content": [media_block]},
            {"role": "assistant", "content": [{"type": "text", "value": "ok"}]},
        ]
        rendered = self.render_messages(messages)

        mm_type_ids = rendered.get("mm_token_type_ids")
        if not mm_type_ids or target not in mm_type_ids or presence_key not in rendered:
            raise RuntimeError(f"Processor did not emit {modality} placeholder tokens for the dummy sample.")

        positions = [i for i, t in enumerate(mm_type_ids) if t == target]
        # Include the surrounding start/end delimiters (vision_start/end or audio_bos/eos) so the
        # fragment matches exactly what the template emits around real media.
        lo = max(positions[0] - 1, 0)
        hi = min(positions[-1] + 2, len(rendered["input_ids"]))

        fragment: dict = {
            "input_ids": list(rendered["input_ids"][lo:hi]),
            "mm_token_type_ids": list(mm_type_ids[lo:hi]),
        }

        for key in _MULTIMODAL_PASSTHROUGH_KEYS:
            if key in rendered:
                fragment[key] = rendered[key]

        self._dummy_fragments[modality] = fragment
        return fragment

View on GitHub (pinned to f28afaf635)

Solutions

1. Verify processor() output includes 'mm_token_type_ids' and 'pixel_values'/'input_features' for a single-media conversation
2. Upgrade transformers to a version whose processor emits mm_token_type_ids for this model family
3. For audio, ensure the feature extractor accepts the 0.1s dummy clip (check min_length config)
4. If the processor genuinely cannot emit these keys, use the NORMAL batching strategy without dummy fragments or fall back to v0

Example fix

# diagnostics before the call
out = processor(text="<image>", images=dummy_img, return_tensors=None)
assert "mm_token_type_ids" in out and "pixel_values" in out, "processor lacks v1 mm tagging; upgrade transformers"

def processor_emits_mm_tags(processor) -> bool:
    out = processor(text="a picture:", images=PIL.Image.new("RGB", (64, 64)))
    return "mm_token_type_ids" in out and "pixel_values" in out

try:
    frag = renderer.get_dummy_media_fragment("image")
except RuntimeError as e:
    if "did not emit" in str(e):
        fall_back_to_normal_batching_or_v0()

When it happens

Trigger: Calling get_dummy_media_fragment on a multimodal processor whose __call__ does not return 'mm_token_type_ids' (or uses different type-id conventions), or whose dummy sample fails to produce pixel_values/input_features (e.g. zero-length audio, misconfigured feature extractor).

Common situations: Wiring a new or exotic multimodal processor into the v1 pipeline; processors from older transformers versions that lack mm_token_type_ids support; audio processors where the dummy 0.1s clip is below the minimum length.

Related errors

• Processor was not found, please check and update your model
• Unsupported torch version detected: torch 2.9.x with Conv3D.
• Unsupported dummy media modality: {modality!r} (expected ima
• Cannot build a dummy media fragment for a text-only processo
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/87d9964ed88ad2f7. Report an issue: GitHub.