{"record":{"id":"d32619a4b6db3ba0","repo":"sgl-project/sglang","slug":"this-use-case-is-not-supported-for-openai-chat-mo","errorCode":null,"errorMessage":"This use case is not supported. For OpenAI chat models, sgl.gen must be right after sgl.assistant","messagePattern":"This use case is not supported\\. For OpenAI chat models, sgl\\.gen must be right after sgl\\.assistant","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/lang/backend/openai.py","lineNumber":291,"sourceCode":"            s.text_ += term[\"text\"]\n            name = term[\"name\"]\n            if name is not None:\n                s.variables[name] = term[\"text\"]\n                s.meta_info[name] = {}\n                s.variable_event[name].set()\n\n        self.spec_kwargs = {}\n        self.spec_format = []\n\n    def generate_stream(\n        self,\n        s: StreamExecutor,\n        sampling_params: SglSamplingParams,\n    ):\n        if sampling_params.dtype is None:\n            if self.is_chat_model:\n                if not s.text_.endswith(self.chat_prefix):\n                    raise RuntimeError(\n                        \"This use case is not supported. \"\n                        \"For OpenAI chat models, sgl.gen must be right after sgl.assistant\"\n                    )\n                prompt = s.messages_\n            else:\n                prompt = s.text_\n\n            kwargs = sampling_params.to_openai_kwargs()\n            generator = openai_completion_stream(\n                client=self.client,\n                token_usage=self.token_usage,\n                is_chat=self.is_chat_model,\n                model=self.model_name,\n                prompt=prompt,\n                **kwargs,\n            )\n            return generator\n        else:","sourceCodeStart":273,"sourceCodeEnd":309,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/lang/backend/openai.py#L273-L309","documentation":"Streaming twin of error 731: for OpenAI chat models, generate_stream requires that sgl.gen directly follow sgl.assistant. When the accumulated prompt doesn't end with the chat prefix, the position of gen is invalid for chat-model streaming and a RuntimeError is raised (this path has no api-spec-execution escape hatch).","triggerScenarios":"Running a program with streaming=True on an OpenAI chat model where sgl.gen appears anywhere other than immediately after sgl.assistant (e.g. gen after a user message or in the middle of a template).","commonSituations":"Adapting completion-model programs to chat models with stream=True; multi-step programs that append user turns after generation started.","solutions":["Move sgl.gen so it immediately follows sgl.assistant in the program.","Use a completion-style (non-chat) model like gpt-3.5-turbo-instruct, where gen may append to any text.","Or disable streaming on that call so the non-streaming path with num_api_spec_tokens can apply."],"exampleFix":"# before\ns += sgl.user(\"Q\")\ns += sgl.gen(\"a\", stream=True)  # chat model, wrong position\n\n# after\ns += sgl.user(\"Q\")\ns += sgl.assistant_sbegin()\ns += sgl.gen(\"a\", stream=True)","handlingStrategy":"validation","validationCode":"if backend.is_chat_model and stream:\n    assert last_role == \"assistant\", \"streaming gen on chat models must follow sgl.assistant\"","typeGuard":"def stream_safe_on_chat(backend, gen_after_assistant: bool) -> bool:\n    return (not backend.is_chat_model) or gen_after_assistant","tryCatchPattern":"try:\n    program.run(backend, stream=True)\nexcept RuntimeError as e:\n    if \"must be right after sgl.assistant\" in str(e):\n        program.run(backend, stream=False)\n    else:\n        raise","preventionTips":["Use assistant_sbegin + gen as a fixed idiom on chat models.","Prefer completion models for free-form streaming programs."],"tags":["frontend","openai","chat-model","streaming","program-structure","sglang"],"backgroundTag":"invalid-chat-template-position","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}