sgl-project/sglang · error · RuntimeError

This use case is not supported. For OpenAI chat models, sgl.

Error message

This use case is not supported. For OpenAI chat models, sgl.gen must be right after sgl.assistant

What it means

Streaming twin of error 731: for OpenAI chat models, generate_stream requires that sgl.gen directly follow sgl.assistant. When the accumulated prompt doesn't end with the chat prefix, the position of gen is invalid for chat-model streaming and a RuntimeError is raised (this path has no api-spec-execution escape hatch).

Source

Thrown at python/sglang/lang/backend/openai.py:291

            s.text_ += term["text"]
            name = term["name"]
            if name is not None:
                s.variables[name] = term["text"]
                s.meta_info[name] = {}
                s.variable_event[name].set()

        self.spec_kwargs = {}
        self.spec_format = []

    def generate_stream(
        self,
        s: StreamExecutor,
        sampling_params: SglSamplingParams,
    ):
        if sampling_params.dtype is None:
            if self.is_chat_model:
                if not s.text_.endswith(self.chat_prefix):
                    raise RuntimeError(
                        "This use case is not supported. "
                        "For OpenAI chat models, sgl.gen must be right after sgl.assistant"
                    )
                prompt = s.messages_
            else:
                prompt = s.text_

            kwargs = sampling_params.to_openai_kwargs()
            generator = openai_completion_stream(
                client=self.client,
                token_usage=self.token_usage,
                is_chat=self.is_chat_model,
                model=self.model_name,
                prompt=prompt,
                **kwargs,
            )
            return generator
        else:

View on GitHub (pinned to 0132848349)

Solutions

Move sgl.gen so it immediately follows sgl.assistant in the program.
Use a completion-style (non-chat) model like gpt-3.5-turbo-instruct, where gen may append to any text.
Or disable streaming on that call so the non-streaming path with num_api_spec_tokens can apply.

Example fix

# before
s += sgl.user("Q")
s += sgl.gen("a", stream=True)  # chat model, wrong position

# after
s += sgl.user("Q")
s += sgl.assistant_sbegin()
s += sgl.gen("a", stream=True)

Defensive patterns

Strategy: validation

Validate before calling

if backend.is_chat_model and stream:
    assert last_role == "assistant", "streaming gen on chat models must follow sgl.assistant"

Type guard

def stream_safe_on_chat(backend, gen_after_assistant: bool) -> bool:
    return (not backend.is_chat_model) or gen_after_assistant

Try / catch

try:
    program.run(backend, stream=True)
except RuntimeError as e:
    if "must be right after sgl.assistant" in str(e):
        program.run(backend, stream=False)
    else:
        raise

Prevention

Use assistant_sbegin + gen as a fixed idiom on chat models.
Prefer completion models for free-form streaming programs.

When it happens

Trigger: Running a program with streaming=True on an OpenAI chat model where sgl.gen appears anywhere other than immediately after sgl.assistant (e.g. gen after a user message or in the middle of a template).

Common situations: Adapting completion-model programs to chat models with stream=True; multi-step programs that append user turns after generation started.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d32619a4b6db3ba0. Report an issue: GitHub.