sgl-project/sglang · error · ValueError

Batch tokenization is not needed for input_embeds. Do not se

Error message

Batch tokenization is not needed for input_embeds. Do not set `enable_tokenizer_batch_encode`.

What it means

Raised when enable_tokenizer_batch_encode is set and a request element supplies input_embeds. Input embeddings bypass tokenization entirely, so batch tokenization is both unnecessary and unsupported for such requests.

Source

Thrown at python/sglang/srt/managers/tokenizer_manager.py:1526

            )
        logger.debug(f"Completed batch processing for {batch_size} requests")
        return tokenized_objs

    def _validate_batch_tokenization_constraints(
        self, batch_size: int, obj: Union[GenerateReqInput, EmbeddingReqInput]
    ) -> None:
        """Validate constraints for batch tokenization processing."""
        for i in range(batch_size):
            if self.is_generation and obj[i].contains_mm_input():
                raise ValueError(
                    "For multimodal input processing do not set `enable_tokenizer_batch_encode`."
                )
            if obj[i].input_ids is not None:
                raise ValueError(
                    "Batch tokenization is not needed for pre-tokenized input_ids. Do not set `enable_tokenizer_batch_encode`."
                )
            if obj[i].input_embeds is not None:
                raise ValueError(
                    "Batch tokenization is not needed for input_embeds. Do not set `enable_tokenizer_batch_encode`."
                )

    def _batch_has_text(
        self, batch_size: int, obj: Union[GenerateReqInput, EmbeddingReqInput]
    ) -> bool:
        """Check if any request in the batch contains text input."""
        for i in range(batch_size):
            if obj[i].text:
                return True
            elif self.is_generation and obj[i].contains_mm_input():
                return True

        return False

    def _should_use_batch_tokenization(self, batch_size, requests) -> bool:
        """Return True if we should run the tokenizer in batch mode.

View on GitHub (pinned to 0132848349)

Solutions

1. Disable enable_tokenizer_batch_encode for deployments that accept input_embeds
2. Send input_embeds requests to a server without the flag
3. Convert to text/input_ids flow if batch encode throughput is required

Example fix

# before
GenerateReqInput(input_embeds=embs, ...)  # with --enable-tokenizer-batch-encode
# after
# launch server without --enable-tokenizer-batch-encode

if any(getattr(r, 'input_embeds', None) is not None for r in batch_requests):
    assert not server_args.enable_tokenizer_batch_encode, 'input_embeds incompatible with batch encode'

except ValueError as e: if 'input_embeds' in str(e): disable the flag and resend

When it happens

Trigger: Server started with --enable-tokenizer-batch-encode and a batch GenerateReqInput element has input_embeds != None.

Common situations: Clients doing custom embedding injection (e.g. retrieval-augmented or phoneme-embedding pipelines) against a server tuned with batch tokenization enabled.

Related errors

• For multimodal input processing do not set `enable_tokenizer
• Batch tokenization is not needed for pre-tokenized input_ids
• {selection_error}{component_suffix}
• Cosmos3CausalAttention requires num_attention_heads divisibl
• VLA action expert should not share the prefix TP layout. Use

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7c92b4c747344bc1. Report an issue: GitHub.