{"record":{"id":"7c92b4c747344bc1","repo":"sgl-project/sglang","slug":"batch-tokenization-is-not-needed-for-input-embeds","errorCode":null,"errorMessage":"Batch tokenization is not needed for input_embeds. Do not set `enable_tokenizer_batch_encode`.","messagePattern":"Batch tokenization is not needed for input_embeds\\. Do not set `enable_tokenizer_batch_encode`\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/managers/tokenizer_manager.py","lineNumber":1526,"sourceCode":"            )\n        logger.debug(f\"Completed batch processing for {batch_size} requests\")\n        return tokenized_objs\n\n    def _validate_batch_tokenization_constraints(\n        self, batch_size: int, obj: Union[GenerateReqInput, EmbeddingReqInput]\n    ) -> None:\n        \"\"\"Validate constraints for batch tokenization processing.\"\"\"\n        for i in range(batch_size):\n            if self.is_generation and obj[i].contains_mm_input():\n                raise ValueError(\n                    \"For multimodal input processing do not set `enable_tokenizer_batch_encode`.\"\n                )\n            if obj[i].input_ids is not None:\n                raise ValueError(\n                    \"Batch tokenization is not needed for pre-tokenized input_ids. Do not set `enable_tokenizer_batch_encode`.\"\n                )\n            if obj[i].input_embeds is not None:\n                raise ValueError(\n                    \"Batch tokenization is not needed for input_embeds. Do not set `enable_tokenizer_batch_encode`.\"\n                )\n\n    def _batch_has_text(\n        self, batch_size: int, obj: Union[GenerateReqInput, EmbeddingReqInput]\n    ) -> bool:\n        \"\"\"Check if any request in the batch contains text input.\"\"\"\n        for i in range(batch_size):\n            if obj[i].text:\n                return True\n            elif self.is_generation and obj[i].contains_mm_input():\n                return True\n\n        return False\n\n    def _should_use_batch_tokenization(self, batch_size, requests) -> bool:\n        \"\"\"Return True if we should run the tokenizer in batch mode.\n","sourceCodeStart":1508,"sourceCodeEnd":1544,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/managers/tokenizer_manager.py#L1508-L1544","documentation":"Raised when enable_tokenizer_batch_encode is set and a request element supplies input_embeds. Input embeddings bypass tokenization entirely, so batch tokenization is both unnecessary and unsupported for such requests.","triggerScenarios":"Server started with --enable-tokenizer-batch-encode and a batch GenerateReqInput element has input_embeds != None.","commonSituations":"Clients doing custom embedding injection (e.g. retrieval-augmented or phoneme-embedding pipelines) against a server tuned with batch tokenization enabled.","solutions":["Disable enable_tokenizer_batch_encode for deployments that accept input_embeds","Send input_embeds requests to a server without the flag","Convert to text/input_ids flow if batch encode throughput is required"],"exampleFix":"# before\nGenerateReqInput(input_embeds=embs, ...)  # with --enable-tokenizer-batch-encode\n# after\n# launch server without --enable-tokenizer-batch-encode","handlingStrategy":"validation","validationCode":"if any(getattr(r, 'input_embeds', None) is not None for r in batch_requests):\n    assert not server_args.enable_tokenizer_batch_encode, 'input_embeds incompatible with batch encode'","typeGuard":null,"tryCatchPattern":"except ValueError as e: if 'input_embeds' in str(e): disable the flag and resend","preventionTips":["Keep embedding-injection clients on servers without batch encode","Smoke-test the exact request shape after toggling server flags"],"tags":["sglang","config","input-embeds","batch-encode"],"backgroundTag":"incompatible-server-config-option","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}