sgl-project/sglang · error · ValueError

InklingMultimodalProcessor

Error message

InklingMultimodalProcessor: {n_aud_ph} audio placeholder token(s) in input_ids but {len(audio_data)} audio(s) provided; counts must match.

What it means

Companion check to the image one: the count of audio placeholder tokens in input_ids must equal len(audio_data) during Inkling assembly. Mismatches mean the prompt was rendered without the right number of audio placeholders, or the audio list diverged from the messages used to build the prompt.

Solutions

1. Re-render input_ids so every audio sample gets exactly one audio placeholder token
2. Ensure audio filtering/removal happens before prompt construction so counts stay in sync
3. Check that AUDIO_TOKEN_ID is configured (not None) when audio is used

Example fix

# before
prompt = template.render(msgs)            # text only
input_ids = tok(prompt)['input_ids']       # 0 audio tokens, 1 audio file
# after
msgs = insert_audio_placeholders(msgs, n=len(audio_data))
input_ids = tok(template.render(msgs))['input_ids']

n_ph = sum(1 for t in input_ids if t == processor.AUDIO_TOKEN_ID) if processor.AUDIO_TOKEN_ID else 0
assert n_ph == len(audio_data or []), f'{n_ph} audio placeholders vs {len(audio_data)} audio'

def audio_placeholders_match(input_ids: list[int], token_id: int|None, data: list) -> bool:
    n = sum(1 for t in input_ids if t == token_id) if token_id is not None else 0
    return n == len(data or [])

try:
    out = await processor.process_mm_data_async(None, None, request_obj)
except ValueError as e:
    if 'audio placeholder' in str(e):
        request_obj.input_ids = render_with_audio_placeholders(msgs, n=len(audio_data))
        out = await processor.process_mm_data_async(None, None, request_obj)
    else: raise

When it happens

Trigger: Pre-tokenized input_ids rendered with a text-only template while audio_data is non-empty; audio list filtered (e.g. dropping unreadable files) after the prompt was built; AUDIO_TOKEN_ID is None so n_aud_ph=0 while audio was supplied.

Common situations: Chat template that doesn't expand consecutive audio tokens; data loader dropping corrupt audio files after prompt construction; template upgrade changing audio token syntax.

Related errors

• Audio placeholder count does not match audio_data
• InklingMultimodalProcessor
• An exception occurred while loading multimodal data
• audio_cap must be non-negative, got
• audio must be a 2D tensor, but got

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/799b3a6a9984f5ba. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/multimodal/processors/inkling.py:215

        # One placeholder per media item (expanded below); a count mismatch (incl. a
        # None token id absent from config) must fail loudly, not drop media silently.
        n_img_ph = (
            sum(1 for t in input_ids if t == self.IMAGE_TOKEN_ID)
            if self.IMAGE_TOKEN_ID is not None
            else 0
        )
        n_aud_ph = (
            sum(1 for t in input_ids if t == self.AUDIO_TOKEN_ID)
            if self.AUDIO_TOKEN_ID is not None
            else 0
        )
        if n_img_ph != len(image_data):
            raise ValueError(
                f"InklingMultimodalProcessor: {n_img_ph} image placeholder token(s) in "
                f"input_ids but {len(image_data)} image(s) provided; counts must match."
            )
        if n_aud_ph != len(audio_data):
            raise ValueError(
                f"InklingMultimodalProcessor: {n_aud_ph} audio placeholder token(s) in "
                f"input_ids but {len(audio_data)} audio(s) provided; counts must match."
            )

        img_feat = (
            self.inkling_processor.process_images(image_data) if image_data else None
        )
        aud_feat = (
            self.inkling_processor.process_audios(audio_data) if audio_data else None
        )

        # Rust processor returns content_hashes; original processor does not.
        img_hashes = img_feat.get("content_hashes") if img_feat else None

        out_ids: List[int] = []
        image_items: List[Tuple[int, int, torch.Tensor]] = []  # (start, end, feature)
        audio_items: List[Tuple[int, int, torch.Tensor]] = []
        i_img = i_aud = 0

View on GitHub (pinned to 0132848349)