{"record":{"id":"d460de83d1a08d54","repo":"hiyouga/LlamaFactory","slug":"sglang-only-supports-n-1","errorCode":null,"errorMessage":"SGLang only supports n=1.","messagePattern":"SGLang only supports n=1\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/chat/sglang_engine.py","lineNumber":178,"sourceCode":"            messages, images or [], videos or [], audios or [], self.processor\n        )\n        paired_messages = messages + [{\"role\": \"assistant\", \"content\": \"\"}]\n        prompt_ids, _ = self.template.encode_oneturn(self.tokenizer, paired_messages, system, tools)\n        prompt_length = len(prompt_ids)\n\n        temperature: Optional[float] = input_kwargs.pop(\"temperature\", None)\n        top_p: Optional[float] = input_kwargs.pop(\"top_p\", None)\n        top_k: Optional[float] = input_kwargs.pop(\"top_k\", None)\n        num_return_sequences: int = input_kwargs.pop(\"num_return_sequences\", 1)\n        repetition_penalty: Optional[float] = input_kwargs.pop(\"repetition_penalty\", None)\n        skip_special_tokens: Optional[bool] = input_kwargs.pop(\"skip_special_tokens\", None)\n        max_length: Optional[int] = input_kwargs.pop(\"max_length\", None)\n        max_new_tokens: Optional[int] = input_kwargs.pop(\"max_new_tokens\", None)\n        seed: Optional[int] = input_kwargs.pop(\"seed\", None)\n        stop: Optional[Union[str, list[str]]] = input_kwargs.pop(\"stop\", None)\n\n        if num_return_sequences != 1:\n            raise NotImplementedError(\"SGLang only supports n=1.\")\n\n        if \"max_new_tokens\" in self.generating_args:\n            max_tokens = self.generating_args[\"max_new_tokens\"]\n        elif \"max_length\" in self.generating_args:\n            if self.generating_args[\"max_length\"] > prompt_length:\n                max_tokens = self.generating_args[\"max_length\"] - prompt_length\n            else:\n                max_tokens = 1\n\n        if max_length:\n            max_tokens = max_length - prompt_length if max_length > prompt_length else 1\n\n        if max_new_tokens:\n            max_tokens = max_new_tokens\n\n        sampling_params = {\n            \"temperature\": temperature if temperature is not None else self.generating_args[\"temperature\"],\n            \"top_p\": (top_p if top_p is not None else self.generating_args[\"top_p\"]) or 1.0,  # top_p must > 0","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/chat/sglang_engine.py#L160-L196","documentation":"SGLangEngine._generate raises NotImplementedError when the caller passes num_return_sequences != 1. The engine pops 'num_return_sequences' from input_kwargs (default 1) and hard-rejects any other value because the SGLang server request path here builds a single-sequence sampling_params payload with no n parameter.","triggerScenarios":"Calling chat/stream_chat/AsyncEngine APIs on an SGLang-backed engine with input_kwargs containing num_return_sequences=2 or more (e.g. benchmark scripts or best-of-n sampling loops that work on the HF or vLLM engine).","commonSituations":"Porting an eval/benchmark script from vllm_engine or hf_engine that uses num_return_sequences>1 for pass@k evaluation; KTO/RM data generation pipelines that sample multiple responses.","solutions":["Remove num_return_sequences from the kwargs passed to the sglang engine and instead call generate repeatedly in a loop, collecting one sequence per call.","Switch engine backend to vllm or hf if multi-sample generation is a hard requirement.","Use per-request temperature/seed variation across looped calls to retain diversity."],"exampleFix":"# before\nresults = await engine.chat(messages, num_return_sequences=4)\n\n# after\nresults = []\nfor i in range(4):\n    results.append(await engine.chat(messages, seed=base_seed + i))","handlingStrategy":"validation","validationCode":"input_kwargs = {k: v for k, v in input_kwargs.items() if not (k == \"num_return_sequences\" and v != 1)}\nif input_kwargs.get(\"num_return_sequences\", 1) != 1 and engine_is_sglang:\n    raise ValueError(\"loop the call instead of n>1 on sglang\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Centralize engine capability checks (supported kwargs per backend) instead of scattering them at call sites.","For pass@k style evals, write an engine-agnostic loop-sampler helper."],"tags":["sglang","sampling","not-implemented","inference"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}