hiyouga/LlamaFactory · error · NotImplementedError

SGLang only supports n=1.

Error message

SGLang only supports n=1.

What it means

SGLangEngine._generate raises NotImplementedError when the caller passes num_return_sequences != 1. The engine pops 'num_return_sequences' from input_kwargs (default 1) and hard-rejects any other value because the SGLang server request path here builds a single-sequence sampling_params payload with no n parameter.

Source

Thrown at src/llamafactory/chat/sglang_engine.py:178

            messages, images or [], videos or [], audios or [], self.processor
        )
        paired_messages = messages + [{"role": "assistant", "content": ""}]
        prompt_ids, _ = self.template.encode_oneturn(self.tokenizer, paired_messages, system, tools)
        prompt_length = len(prompt_ids)

        temperature: Optional[float] = input_kwargs.pop("temperature", None)
        top_p: Optional[float] = input_kwargs.pop("top_p", None)
        top_k: Optional[float] = input_kwargs.pop("top_k", None)
        num_return_sequences: int = input_kwargs.pop("num_return_sequences", 1)
        repetition_penalty: Optional[float] = input_kwargs.pop("repetition_penalty", None)
        skip_special_tokens: Optional[bool] = input_kwargs.pop("skip_special_tokens", None)
        max_length: Optional[int] = input_kwargs.pop("max_length", None)
        max_new_tokens: Optional[int] = input_kwargs.pop("max_new_tokens", None)
        seed: Optional[int] = input_kwargs.pop("seed", None)
        stop: Optional[Union[str, list[str]]] = input_kwargs.pop("stop", None)

        if num_return_sequences != 1:
            raise NotImplementedError("SGLang only supports n=1.")

        if "max_new_tokens" in self.generating_args:
            max_tokens = self.generating_args["max_new_tokens"]
        elif "max_length" in self.generating_args:
            if self.generating_args["max_length"] > prompt_length:
                max_tokens = self.generating_args["max_length"] - prompt_length
            else:
                max_tokens = 1

        if max_length:
            max_tokens = max_length - prompt_length if max_length > prompt_length else 1

        if max_new_tokens:
            max_tokens = max_new_tokens

        sampling_params = {
            "temperature": temperature if temperature is not None else self.generating_args["temperature"],
            "top_p": (top_p if top_p is not None else self.generating_args["top_p"]) or 1.0,  # top_p must > 0

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove num_return_sequences from the kwargs passed to the sglang engine and instead call generate repeatedly in a loop, collecting one sequence per call.
2. Switch engine backend to vllm or hf if multi-sample generation is a hard requirement.
3. Use per-request temperature/seed variation across looped calls to retain diversity.

Example fix

# before
results = await engine.chat(messages, num_return_sequences=4)

# after
results = []
for i in range(4):
    results.append(await engine.chat(messages, seed=base_seed + i))

input_kwargs = {k: v for k, v in input_kwargs.items() if not (k == "num_return_sequences" and v != 1)}
if input_kwargs.get("num_return_sequences", 1) != 1 and engine_is_sglang:
    raise ValueError("loop the call instead of n>1 on sglang")

When it happens

Trigger: Calling chat/stream_chat/AsyncEngine APIs on an SGLang-backed engine with input_kwargs containing num_return_sequences=2 or more (e.g. benchmark scripts or best-of-n sampling loops that work on the HF or vLLM engine).

Common situations: Porting an eval/benchmark script from vllm_engine or hf_engine that uses num_return_sequences>1 for pass@k evaluation; KTO/RM data generation pipelines that sample multiple responses.

Related errors

• SGLang engine does not support `get_scores`.
• SGLang not install, you may need to run `pip install sglang[
• SGLang server initialization failed: {str(e)}.
• SGLang server error: {response.status_code}, {response.text}
• vLLM engine does not support `get_scores`.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/d460de83d1a08d54. Report an issue: GitHub.