sgl-project/sglang · error · RuntimeError

This use case is not supported if api speculative execution

Error message

This use case is not supported if api speculative execution is off. For OpenAI chat models, sgl.gen must be right after sgl.assistant. Example of adding api speculative execution: @function(num_api_spec_tokens=128).

What it means

For OpenAI chat models, sgl.gen must immediately follow sgl.assistant so the library can map generation onto the chat API. If API speculative execution is off (no num_api_spec_tokens on @function) and the accumulated text does not end with the chat prefix, this invariant is violated and generate() raises RuntimeError with guidance to enable speculative tokens.

Source

Thrown at python/sglang/lang/backend/openai.py:150

                assert (
                    value == self.spec_kwargs[key]
                ), "sampling parameters should be consistent if turn on api speculative execution."
        self.spec_format.append(
            {"text": "", "stop": params["stop"], "name": spec_var_name}
        )
        return "", {}

    def generate(
        self,
        s: StreamExecutor,
        sampling_params: SglSamplingParams,
        spec_var_name: str = None,
    ):
        if sampling_params.dtype is None:
            if self.is_chat_model:
                if s.num_api_spec_tokens is None:
                    if not s.text_.endswith(self.chat_prefix):
                        raise RuntimeError(
                            "This use case is not supported if api speculative execution is off. "
                            "For OpenAI chat models, sgl.gen must be right after sgl.assistant. "
                            "Example of adding api speculative execution: @function(num_api_spec_tokens=128)."
                        )
                    prompt = s.messages_
                else:
                    return self._prepare_spec_execution(
                        sampling_params, s.num_api_spec_tokens, spec_var_name
                    )
            else:
                prompt = s.text_

            kwargs = sampling_params.to_openai_kwargs()
            if (
                self.model_name.startswith("o1")
                or self.model_name.startswith("o3")
                or "o1" in self.model_name
            ):

View on GitHub (pinned to 0132848349)

Solutions

Add API speculative execution: @function(num_api_spec_tokens=128) so gen can be placed more freely.
Restructure the program so sgl.gen comes immediately after sgl.assistant.
Use a non-chat (completion) model such as gpt-3.5-turbo-instruct where any suffix position is allowed.

Example fix

# before
@function
def demo(s):
    s += sgl.user("Q")
    s += sgl.gen("a")  # not after assistant -> error on chat models

# after
@function(num_api_spec_tokens=128)
def demo(s):
    s += sgl.user("Q")
    s += sgl.assistant_sbegin()
    s += sgl.gen("a")

Defensive patterns

Strategy: validation

Validate before calling

from sglang.lang.ir import SglGen
# ensure gen node's parent is assistant before running on OpenAI chat backend
if backend.is_chat_model and num_api_spec_tokens is None:
    assert last_role == "assistant", "sgl.gen must follow sgl.assistant on chat models"

Type guard

def chat_safe(backend, program_has_spec_tokens: bool) -> bool:
    return (not backend.is_chat_model) or program_has_spec_tokens or gen_follows_assistant

Try / catch

try:
    program.run(backend)
except RuntimeError as e:
    if "speculative execution" in str(e):
        rerun_with(num_api_spec_tokens=128)
    else:
        raise

Prevention

Always decorate chat-model programs with @function(num_api_spec_tokens=128).
Keep gen directly after assistant_sbegin in chat templates.
Smoke-test programs on the exact backend class you'll deploy.

When it happens

Trigger: Calling sgl.gen somewhere other than directly after sgl.assistant (e.g. gen after system/user message, or nested/branched programs) on an OpenAI chat model backend without @function(num_api_spec_tokens=...).

Common situations: Porting programs written for non-chat backends to gpt-3.5-turbo/gpt-4; multi-turn templates with gen not adjacent to the assistant turn; forgetting the decorator argument.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/055d24a47aa5536b. Report an issue: GitHub.