{"record":{"id":"055d24a47aa5536b","repo":"sgl-project/sglang","slug":"this-use-case-is-not-supported-if-api-speculative","errorCode":null,"errorMessage":"This use case is not supported if api speculative execution is off. For OpenAI chat models, sgl.gen must be right after sgl.assistant. Example of adding api speculative execution: @function(num_api_spec_tokens=128).","messagePattern":"This use case is not supported if api speculative execution is off\\. For OpenAI chat models, sgl\\.gen must be right after sgl\\.assistant\\. Example of adding api speculative execution: @function\\(num_api_spec_tokens=128\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/lang/backend/openai.py","lineNumber":150,"sourceCode":"                assert (\n                    value == self.spec_kwargs[key]\n                ), \"sampling parameters should be consistent if turn on api speculative execution.\"\n        self.spec_format.append(\n            {\"text\": \"\", \"stop\": params[\"stop\"], \"name\": spec_var_name}\n        )\n        return \"\", {}\n\n    def generate(\n        self,\n        s: StreamExecutor,\n        sampling_params: SglSamplingParams,\n        spec_var_name: str = None,\n    ):\n        if sampling_params.dtype is None:\n            if self.is_chat_model:\n                if s.num_api_spec_tokens is None:\n                    if not s.text_.endswith(self.chat_prefix):\n                        raise RuntimeError(\n                            \"This use case is not supported if api speculative execution is off. \"\n                            \"For OpenAI chat models, sgl.gen must be right after sgl.assistant. \"\n                            \"Example of adding api speculative execution: @function(num_api_spec_tokens=128).\"\n                        )\n                    prompt = s.messages_\n                else:\n                    return self._prepare_spec_execution(\n                        sampling_params, s.num_api_spec_tokens, spec_var_name\n                    )\n            else:\n                prompt = s.text_\n\n            kwargs = sampling_params.to_openai_kwargs()\n            if (\n                self.model_name.startswith(\"o1\")\n                or self.model_name.startswith(\"o3\")\n                or \"o1\" in self.model_name\n            ):","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/lang/backend/openai.py#L132-L168","documentation":"For OpenAI chat models, sgl.gen must immediately follow sgl.assistant so the library can map generation onto the chat API. If API speculative execution is off (no num_api_spec_tokens on @function) and the accumulated text does not end with the chat prefix, this invariant is violated and generate() raises RuntimeError with guidance to enable speculative tokens.","triggerScenarios":"Calling sgl.gen somewhere other than directly after sgl.assistant (e.g. gen after system/user message, or nested/branched programs) on an OpenAI chat model backend without @function(num_api_spec_tokens=...).","commonSituations":"Porting programs written for non-chat backends to gpt-3.5-turbo/gpt-4; multi-turn templates with gen not adjacent to the assistant turn; forgetting the decorator argument.","solutions":["Add API speculative execution: @function(num_api_spec_tokens=128) so gen can be placed more freely.","Restructure the program so sgl.gen comes immediately after sgl.assistant.","Use a non-chat (completion) model such as gpt-3.5-turbo-instruct where any suffix position is allowed."],"exampleFix":"# before\n@function\ndef demo(s):\n    s += sgl.user(\"Q\")\n    s += sgl.gen(\"a\")  # not after assistant -> error on chat models\n\n# after\n@function(num_api_spec_tokens=128)\ndef demo(s):\n    s += sgl.user(\"Q\")\n    s += sgl.assistant_sbegin()\n    s += sgl.gen(\"a\")","handlingStrategy":"validation","validationCode":"from sglang.lang.ir import SglGen\n# ensure gen node's parent is assistant before running on OpenAI chat backend\nif backend.is_chat_model and num_api_spec_tokens is None:\n    assert last_role == \"assistant\", \"sgl.gen must follow sgl.assistant on chat models\"","typeGuard":"def chat_safe(backend, program_has_spec_tokens: bool) -> bool:\n    return (not backend.is_chat_model) or program_has_spec_tokens or gen_follows_assistant","tryCatchPattern":"try:\n    program.run(backend)\nexcept RuntimeError as e:\n    if \"speculative execution\" in str(e):\n        rerun_with(num_api_spec_tokens=128)\n    else:\n        raise","preventionTips":["Always decorate chat-model programs with @function(num_api_spec_tokens=128).","Keep gen directly after assistant_sbegin in chat templates.","Smoke-test programs on the exact backend class you'll deploy."],"tags":["frontend","openai","chat-model","program-structure","sglang"],"backgroundTag":"invalid-chat-template-position","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}