hiyouga/LlamaFactory · error · HTTPException

Cannot stream function calls.

Error message

Cannot stream function calls.

What it means

Raised as HTTP 400 by the streaming endpoint (stream: true) when the request carries a non-empty tools list. LlamaFactory's stream path does not implement tool-call streaming, so function-calling requests must use the non-streaming endpoint. The check happens after _process_request, before any chunk is yielded.

Source

Thrown at src/llamafactory/api/chat.py:253

        prompt_length = response.prompt_length
        response_length += response.response_length

    usage = ChatCompletionResponseUsage(
        prompt_tokens=prompt_length,
        completion_tokens=response_length,
        total_tokens=prompt_length + response_length,
    )

    return ChatCompletionResponse(id=completion_id, model=request.model, choices=choices, usage=usage)


async def create_stream_chat_completion_response(
    request: "ChatCompletionRequest", chat_model: "ChatModel"
) -> AsyncGenerator[str, None]:
    completion_id = f"chatcmpl-{uuid.uuid4().hex}"
    input_messages, system, tools, images, videos, audios = _process_request(request)
    if tools:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot stream function calls.")

    if request.n > 1:
        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Cannot stream multiple responses.")

    yield _create_stream_chat_completion_chunk(
        completion_id=completion_id, model=request.model, delta=ChatCompletionMessage(role=Role.ASSISTANT, content="")
    )
    async for new_token in chat_model.astream_chat(
        input_messages,
        system,
        tools,
        images,
        videos,
        audios,
        do_sample=request.do_sample,
        temperature=request.temperature,
        top_p=request.top_p,
        max_new_tokens=request.max_tokens,

View on GitHub (pinned to f28afaf635)

Solutions

1. Set stream: false when sending tools.
2. Or drop the tools array if function calling is not actually needed for this request.
3. Wrap the streaming call and fall back to non-streaming when this 400 is returned.
4. Track LlamaFactory releases for tool-call streaming support before re-enabling.

Example fix

// before
const res = await client.chat.completions.create({ model, messages, tools, stream: true });
// after
const res = await client.chat.completions.create({ model, messages, tools, stream: false });

def can_stream(request):
    return not (request.get("tools") and request.get("stream"))

if not can_stream(payload):
    payload = {**payload, "stream": False}

const canStream = (req) => !req.tools?.length || !req.stream;

try { stream(...) } catch (e) { if (e.status === 400 && e.detail === 'Cannot stream function calls.') { return nonStream({...req, stream: false}); } throw e; }

When it happens

Trigger: POST /v1/chat/completions with stream: true and a non-empty tools array; client SDKs (e.g. openai-python with stream=True + tools) that always set both.

Common situations: Copy-pasting an OpenAI tool-use streaming example against LlamaFactory; toggling stream=true for latency while forgetting tools are set on the client.

Related errors

• Invalid tools
• Cannot stream multiple responses.
• Invalid input type {input_item.type}.
• Invalid request
• tools is not valid JSON: {tools!r}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/38f8df20ba478ef2. Report an issue: GitHub.