hiyouga/LlamaFactory · error · HTTPException

Cannot stream multiple responses.

Error message

Cannot stream multiple responses.

What it means

Raised as HTTP 400 by the streaming endpoint when request.n > 1. The streaming implementation yields a single completion choice and cannot fan out n samples; only n=1 is supported with stream: true.

Source

Thrown at src/llamafactory/api/chat.py:256

    usage = ChatCompletionResponseUsage(
        prompt_tokens=prompt_length,
        completion_tokens=response_length,
        total_tokens=prompt_length + response_length,
    )

    return ChatCompletionResponse(id=completion_id, model=request.model, choices=choices, usage=usage)


async def create_stream_chat_completion_response(
    request: "ChatCompletionRequest", chat_model: "ChatModel"
) -> AsyncGenerator[str, None]:
    completion_id = f"chatcmpl-{uuid.uuid4().hex}"
    input_messages, system, tools, images, videos, audios = _process_request(request)
    if tools:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot stream function calls.")

    if request.n > 1:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot stream multiple responses.")

    yield _create_stream_chat_completion_chunk(
        completion_id=completion_id, model=request.model, delta=ChatCompletionMessage(role=Role.ASSISTANT, content="")
    )
    async for new_token in chat_model.astream_chat(
        input_messages,
        system,
        tools,
        images,
        videos,
        audios,
        do_sample=request.do_sample,
        temperature=request.temperature,
        top_p=request.top_p,
        max_new_tokens=request.max_tokens,
        repetition_penalty=request.presence_penalty,
        stop=request.stop,
    ):

View on GitHub (pinned to f28afaf635)

Solutions

1. Set n: 1 (or omit n) when stream: true.
2. If you need multiple samples, either use stream: false with n>1, or issue n parallel streaming requests with n=1 each.
3. Disable the multi-response option in your client UI when streaming is on.

Example fix

// before
{ model, messages, stream: true, n: 3 }
// after
{ model, messages, stream: true, n: 1 }

def stream_ok(payload):
    return not (payload.get("stream") and (payload.get("n") or 1) > 1)

const streamOk = (req) => !req.stream || (req.n ?? 1) === 1;

catch (e) { if (e.status === 400 && e.detail === 'Cannot stream multiple responses.') { return nonStream({...req, stream: false, n: req.n}); } throw e; }

When it happens

Trigger: POST /v1/chat/completions with stream: true and n: 2+; client SDKs that default n or allow best_of-style sampling; benchmark scripts sampling multiple continuations.

Common situations: Porting OpenAI n>1 sampling code to LlamaFactory; UIs exposing a 'responses' count control combined with streaming.

Related errors

• Cannot stream function calls.
• Invalid input type {input_item.type}.
• Invalid tools
• Invalid request
• `max_samples` is incompatible with `streaming`.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/12bad244bf886c5a. Report an issue: GitHub.