hiyouga/LlamaFactory · error · ValueError

The current model does not support `stream_chat`.

Error message

The current model does not support `stream_chat`.

What it means

ValueError raised by HuggingfaceEngine.stream_chat when the loaded model cannot generate (can_generate false — a scoring/classification checkpoint). Identical root cause to the non-streaming chat variant: generative streaming requested from a non-generative model.

Source

Thrown at src/llamafactory/chat/hf_engine.py:388

            audios,
            input_kwargs,
        )
        async with self.semaphore:
            return await asyncio.to_thread(self._chat, *input_args)

    @override
    async def stream_chat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> AsyncGenerator[str, None]:
        if not self.can_generate:
            raise ValueError("The current model does not support `stream_chat`.")

        input_args = (
            self.model,
            self.tokenizer,
            self.processor,
            self.template,
            self.generating_args,
            messages,
            system,
            tools,
            images,
            videos,
            audios,
            input_kwargs,
        )
        async with self.semaphore:
            stream = self._stream_chat(*input_args)
            while True:

View on GitHub (pinned to f28afaf635)

Solutions

1. Load a causal-LM checkpoint for streaming chat.
2. Use get_scores for scoring models.
3. Inspect config.json architectures to confirm the checkpoint type before wiring it into chat flows.

Example fix

# before
chat_model = ChatModel({'model_name_or_path': 'outputs/rm_dir'})
for tok in chat_model.stream_chat([...]): ...
# after
chat_model = ChatModel({'model_name_or_path': 'meta-llama/Llama-3.1-8B-Instruct'})
for tok in chat_model.stream_chat([...]): ...

import json
def is_generative(model_dir):
    archs = json.load(open(f"{model_dir}/config.json"))["architectures"]
    return any("CausalLM" in a or "LMHead" in a for a in archs)

assert is_generative(model_path)  # before streaming

try { async for tok in chat_model.astream_chat(msgs): ... } except ValueError as e: if 'does not support `stream_chat`' in str(e): fall_back_to_non_stream_or_scorer() else: raise

When it happens

Trigger: Calling stream_chat (or /v1/chat/completions with stream: true) against a reward/sequence-classification model loaded via the HF backend.

Common situations: Streaming UI pointed at an RM checkpoint; testing stream endpoints with whatever checkpoint was last exported.

Related errors

• The current model does not support `chat`.
• Cannot get scores using an auto-regressive model.
• The model does not have a submodule named '{submodule_name}'
• Cannot stream function calls.
• Cannot stream multiple responses.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/210ec1aac257a877. Report an issue: GitHub.