{"record":{"id":"87d9964ed88ad2f7","repo":"hiyouga/LlamaFactory","slug":"processor-did-not-emit-modality-placeholder-toke","errorCode":null,"errorMessage":"Processor did not emit {modality} placeholder tokens for the dummy sample.","messagePattern":"Processor did not emit (.+?) placeholder tokens for the dummy sample\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/rendering/rendering.py","lineNumber":243,"sourceCode":"        elif modality == \"video\":\n            # A minimal clip: the temporal patch size is typically 2, so provide two frames.\n            media_block = {\"type\": \"video_url\", \"value\": np.zeros((2, 64, 64, 3), dtype=np.uint8)}\n            target, presence_key = 2, \"pixel_values_videos\"\n        else:\n            # A short synthetic waveform at the model's sampling rate; the feature extractor pads it.\n            sr = self.processor.feature_extractor.sampling_rate\n            media_block = {\"type\": \"audio_url\", \"value\": np.zeros(sr // 10, dtype=np.float32)}\n            target, presence_key = 3, \"input_features\"\n\n        messages: list[Message] = [\n            {\"role\": \"user\", \"content\": [media_block]},\n            {\"role\": \"assistant\", \"content\": [{\"type\": \"text\", \"value\": \"ok\"}]},\n        ]\n        rendered = self.render_messages(messages)\n\n        mm_type_ids = rendered.get(\"mm_token_type_ids\")\n        if not mm_type_ids or target not in mm_type_ids or presence_key not in rendered:\n            raise RuntimeError(f\"Processor did not emit {modality} placeholder tokens for the dummy sample.\")\n\n        positions = [i for i, t in enumerate(mm_type_ids) if t == target]\n        # Include the surrounding start/end delimiters (vision_start/end or audio_bos/eos) so the\n        # fragment matches exactly what the template emits around real media.\n        lo = max(positions[0] - 1, 0)\n        hi = min(positions[-1] + 2, len(rendered[\"input_ids\"]))\n\n        fragment: dict = {\n            \"input_ids\": list(rendered[\"input_ids\"][lo:hi]),\n            \"mm_token_type_ids\": list(mm_type_ids[lo:hi]),\n        }\n\n        for key in _MULTIMODAL_PASSTHROUGH_KEYS:\n            if key in rendered:\n                fragment[key] = rendered[key]\n\n        self._dummy_fragments[modality] = fragment\n        return fragment","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/rendering/rendering.py#L225-L261","documentation":"To build a dummy media fragment, the renderer renders a minimal image/video/audio conversation and expects the processor to emit mm_token_type_ids containing the modality's marker id (1=image, 2=video, 3=audio) plus a presence key like pixel_values/input_features. If the processor does not tag tokens that way (non-conforming or partially supported multimodal processor), fragment extraction cannot proceed and a RuntimeError is raised.","triggerScenarios":"Calling get_dummy_media_fragment on a multimodal processor whose __call__ does not return 'mm_token_type_ids' (or uses different type-id conventions), or whose dummy sample fails to produce pixel_values/input_features (e.g. zero-length audio, misconfigured feature extractor).","commonSituations":"Wiring a new or exotic multimodal processor into the v1 pipeline; processors from older transformers versions that lack mm_token_type_ids support; audio processors where the dummy 0.1s clip is below the minimum length.","solutions":["Verify processor() output includes 'mm_token_type_ids' and 'pixel_values'/'input_features' for a single-media conversation","Upgrade transformers to a version whose processor emits mm_token_type_ids for this model family","For audio, ensure the feature extractor accepts the 0.1s dummy clip (check min_length config)","If the processor genuinely cannot emit these keys, use the NORMAL batching strategy without dummy fragments or fall back to v0"],"exampleFix":"# diagnostics before the call\nout = processor(text=\"<image>\", images=dummy_img, return_tensors=None)\nassert \"mm_token_type_ids\" in out and \"pixel_values\" in out, \"processor lacks v1 mm tagging; upgrade transformers\"","handlingStrategy":"validation","validationCode":"def processor_emits_mm_tags(processor) -> bool:\n    out = processor(text=\"a picture:\", images=PIL.Image.new(\"RGB\", (64, 64)))\n    return \"mm_token_type_ids\" in out and \"pixel_values\" in out","typeGuard":null,"tryCatchPattern":"try:\n    frag = renderer.get_dummy_media_fragment(\"image\")\nexcept RuntimeError as e:\n    if \"did not emit\" in str(e):\n        fall_back_to_normal_batching_or_v0()","preventionTips":["Smoke-test new multimodal processors for mm_token_type_ids before wiring them into v1","Pin a transformers version known to support the processor's mm tagging"],"tags":["multimodal","processor","dummy-data","compatibility","transformers-version"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}