{"record":{"id":"12bad244bf886c5a","repo":"hiyouga/LlamaFactory","slug":"cannot-stream-multiple-responses","errorCode":null,"errorMessage":"Cannot stream multiple responses.","messagePattern":"Cannot stream multiple responses\\.","errorType":"http","errorClass":"HTTPException","httpStatus":400,"severity":"error","filePath":"src/llamafactory/api/chat.py","lineNumber":256,"sourceCode":"    usage = ChatCompletionResponseUsage(\n        prompt_tokens=prompt_length,\n        completion_tokens=response_length,\n        total_tokens=prompt_length + response_length,\n    )\n\n    return ChatCompletionResponse(id=completion_id, model=request.model, choices=choices, usage=usage)\n\n\nasync def create_stream_chat_completion_response(\n    request: \"ChatCompletionRequest\", chat_model: \"ChatModel\"\n) -> AsyncGenerator[str, None]:\n    completion_id = f\"chatcmpl-{uuid.uuid4().hex}\"\n    input_messages, system, tools, images, videos, audios = _process_request(request)\n    if tools:\n        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Cannot stream function calls.\")\n\n    if request.n > 1:\n        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=\"Cannot stream multiple responses.\")\n\n    yield _create_stream_chat_completion_chunk(\n        completion_id=completion_id, model=request.model, delta=ChatCompletionMessage(role=Role.ASSISTANT, content=\"\")\n    )\n    async for new_token in chat_model.astream_chat(\n        input_messages,\n        system,\n        tools,\n        images,\n        videos,\n        audios,\n        do_sample=request.do_sample,\n        temperature=request.temperature,\n        top_p=request.top_p,\n        max_new_tokens=request.max_tokens,\n        repetition_penalty=request.presence_penalty,\n        stop=request.stop,\n    ):","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/api/chat.py#L238-L274","documentation":"Raised as HTTP 400 by the streaming endpoint when request.n > 1. The streaming implementation yields a single completion choice and cannot fan out n samples; only n=1 is supported with stream: true.","triggerScenarios":"POST /v1/chat/completions with stream: true and n: 2+; client SDKs that default n or allow best_of-style sampling; benchmark scripts sampling multiple continuations.","commonSituations":"Porting OpenAI n>1 sampling code to LlamaFactory; UIs exposing a 'responses' count control combined with streaming.","solutions":["Set n: 1 (or omit n) when stream: true.","If you need multiple samples, either use stream: false with n>1, or issue n parallel streaming requests with n=1 each.","Disable the multi-response option in your client UI when streaming is on."],"exampleFix":"// before\n{ model, messages, stream: true, n: 3 }\n// after\n{ model, messages, stream: true, n: 1 }","handlingStrategy":"validation","validationCode":"def stream_ok(payload):\n    return not (payload.get(\"stream\") and (payload.get(\"n\") or 1) > 1)","typeGuard":"const streamOk = (req) => !req.stream || (req.n ?? 1) === 1;","tryCatchPattern":"catch (e) { if (e.status === 400 && e.detail === 'Cannot stream multiple responses.') { return nonStream({...req, stream: false, n: req.n}); } throw e; }","preventionTips":["Default n to 1 in your client wrapper.","For n samples over streaming, run n parallel n=1 requests.","Disable the 'responses' UI control when stream is toggled on."],"tags":["api","streaming","sampling","http-400"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}