sgl-project/sglang · error · NotImplementedError

select/choices is not supported for chat models. Please try

Error message

select/choices is not supported for chat models. Please try to use a non-chat model such as gpt-3.5-turbo-instruct

What it means

The OpenAI backend's select() explicitly raises NotImplementedError for chat models: choice scoring relies on logprobs over supplied choice token sequences, which chat/completions cannot provide. The message directs users to a completion model (e.g. gpt-3.5-turbo-instruct) that exposes logprobs.

Source

Thrown at python/sglang/lang/backend/openai.py:321

                is_chat=self.is_chat_model,
                model=self.model_name,
                prompt=prompt,
                **kwargs,
            )
            return generator
        else:
            raise ValueError(f"Unknown dtype: {sampling_params.dtype}")

    def select(
        self,
        s: StreamExecutor,
        choices: List[str],
        temperature: float,
        choices_method: ChoicesSamplingMethod,
    ) -> ChoicesDecision:
        """Note: `choices_method` is not used by the OpenAI backend."""
        if self.is_chat_model:
            raise NotImplementedError(
                "select/choices is not supported for chat models. "
                "Please try to use a non-chat model such as gpt-3.5-turbo-instruct"
            )

        n_choices = len(choices)
        token_ids = [self.tokenizer.encode(x) for x in choices]
        scores = [0] * n_choices
        valid = [len(x) > 0 for x in token_ids]
        prompt_tokens = self.tokenizer.encode(s.text_)

        max_len = max([len(x) for x in token_ids])
        for step in range(max_len):
            # Build logit bias
            logit_bias = {}
            for i in range(n_choices):
                if valid[i]:
                    logit_bias[token_ids[i][step]] = 100

View on GitHub (pinned to 0132848349)

Solutions

1. Switch to a non-chat model such as gpt-3.5-turbo-instruct for sgl.select.
2. Replace sgl.select with sgl.gen plus prompt-based enumeration of choices and parse the text.
3. Use the RuntimeEndpoint backend (local sglang server), which supports select natively.

Example fix

# before
backend = sgl.OpenAI("gpt-4o")
... sgl.select(x, choices=["yes","no"], temperature=0)

# after
backend = sgl.OpenAI("gpt-3.5-turbo-instruct")
... sgl.select(x, choices=["yes","no"], temperature=0)

if getattr(backend, "is_chat_model", False):
    # replace select with gen-based choice prompt
    pass

def supports_select(backend) -> bool:
    return not getattr(backend, "is_chat_model", False) and \
           type(backend).select is not BaseBackend.select

try:
    decision = backend.select(s, choices, temperature, method)
except NotImplementedError:
    # chat model: enumerate choices in prompt and parse answer
    s += sgl.gen("pick", ...)

When it happens

Trigger: Calling sgl.select(choices=[...]) in a program running on an OpenAI chat model backend (is_chat_model=True), i.e. model names containing 'chat'/'gpt-3.5-turbo'/'gpt-4' style chat endpoints.

Common situations: Porting choice-based workflows (classification, constrained selection) from local sglang or completion APIs to OpenAI chat models.

Related errors

• This use case is not supported if api speculative execution
• This use case is not supported. For OpenAI chat models, sgl.
• Unknown dtype: {sampling_params.dtype}
• Crusoe API key required. Pass api_key= or set CRUSOE_API_KEY
• Invalid dtype: {sampling_params.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/58c1b2a9b4042317. Report an issue: GitHub.