{"record":{"id":"210ec1aac257a877","repo":"hiyouga/LlamaFactory","slug":"the-current-model-does-not-support-stream-chat","errorCode":null,"errorMessage":"The current model does not support `stream_chat`.","messagePattern":"The current model does not support `stream_chat`\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/chat/hf_engine.py","lineNumber":388,"sourceCode":"            audios,\n            input_kwargs,\n        )\n        async with self.semaphore:\n            return await asyncio.to_thread(self._chat, *input_args)\n\n    @override\n    async def stream_chat(\n        self,\n        messages: list[dict[str, str]],\n        system: Optional[str] = None,\n        tools: Optional[str] = None,\n        images: Optional[list[\"ImageInput\"]] = None,\n        videos: Optional[list[\"VideoInput\"]] = None,\n        audios: Optional[list[\"AudioInput\"]] = None,\n        **input_kwargs,\n    ) -> AsyncGenerator[str, None]:\n        if not self.can_generate:\n            raise ValueError(\"The current model does not support `stream_chat`.\")\n\n        input_args = (\n            self.model,\n            self.tokenizer,\n            self.processor,\n            self.template,\n            self.generating_args,\n            messages,\n            system,\n            tools,\n            images,\n            videos,\n            audios,\n            input_kwargs,\n        )\n        async with self.semaphore:\n            stream = self._stream_chat(*input_args)\n            while True:","sourceCodeStart":370,"sourceCodeEnd":406,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/chat/hf_engine.py#L370-L406","documentation":"ValueError raised by HuggingfaceEngine.stream_chat when the loaded model cannot generate (can_generate false — a scoring/classification checkpoint). Identical root cause to the non-streaming chat variant: generative streaming requested from a non-generative model.","triggerScenarios":"Calling stream_chat (or /v1/chat/completions with stream: true) against a reward/sequence-classification model loaded via the HF backend.","commonSituations":"Streaming UI pointed at an RM checkpoint; testing stream endpoints with whatever checkpoint was last exported.","solutions":["Load a causal-LM checkpoint for streaming chat.","Use get_scores for scoring models.","Inspect config.json architectures to confirm the checkpoint type before wiring it into chat flows."],"exampleFix":"# before\nchat_model = ChatModel({'model_name_or_path': 'outputs/rm_dir'})\nfor tok in chat_model.stream_chat([...]): ...\n# after\nchat_model = ChatModel({'model_name_or_path': 'meta-llama/Llama-3.1-8B-Instruct'})\nfor tok in chat_model.stream_chat([...]): ...","handlingStrategy":"validation","validationCode":"import json\ndef is_generative(model_dir):\n    archs = json.load(open(f\"{model_dir}/config.json\"))[\"architectures\"]\n    return any(\"CausalLM\" in a or \"LMHead\" in a for a in archs)\n\nassert is_generative(model_path)  # before streaming","typeGuard":null,"tryCatchPattern":"try { async for tok in chat_model.astream_chat(msgs): ... } except ValueError as e: if 'does not support `stream_chat`' in str(e): fall_back_to_non_stream_or_scorer() else: raise","preventionTips":["Validate the checkpoint type once at startup, not per request.","Route scoring checkpoints to score endpoints only.","Add a can_generate style capability check to your model registry."],"tags":["model-loading","hf-engine","streaming","model-type"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}