hiyouga/LlamaFactory · error · ValueError

The current model does not support `chat`.

Error message

The current model does not support `chat`.

What it means

ValueError raised by HuggingfaceEngine.chat when the loaded model cannot generate (can_generate is false — typically a sequence-classification/RM-style checkpoint). Chat is a generation task; the HF engine routes scoring-only models to get_scores instead.

Source

Thrown at src/llamafactory/chat/hf_engine.py:357

            add_special_tokens=False,
        ).to(device)
        values: torch.Tensor = model(**inputs, return_dict=True, use_cache=False)[-1]
        scores = values.gather(dim=-1, index=(inputs["attention_mask"].sum(dim=-1, keepdim=True) - 1))
        return scores

    @override
    async def chat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> list["Response"]:
        if not self.can_generate:
            raise ValueError("The current model does not support `chat`.")

        input_args = (
            self.model,
            self.tokenizer,
            self.processor,
            self.template,
            self.generating_args,
            messages,
            system,
            tools,
            images,
            videos,
            audios,
            input_kwargs,
        )
        async with self.semaphore:
            return await asyncio.to_thread(self._chat, *input_args)

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a causal-LM checkpoint (AutoModelForCausalLM-compatible) for chat.
2. If the model is a scorer, call get_scores / the score-evaluation endpoint instead of chat.
3. Check config.json architectures in the model dir — ForSequenceClassification indicates a scorer.
4. Re-export or fine-tune with the correct stage (sft) to get a generative model.

Example fix

# before
chat_model = ChatModel({'model_name_or_path': 'outputs/reward_model_dir'})
resp = chat_model.chat([...])
# after
chat_model = ChatModel({'model_name_or_path': 'qwen/Qwen2.5-7B-Instruct'})
resp = chat_model.chat([...])

import json
def is_generative(model_dir):
    archs = json.load(open(f"{model_dir}/config.json"))["architectures"]
    return any("CausalLM" in a or "LMHead" in a for a in archs)

assert is_generative(model_path)

try { await chat_model.achat(msgs) } except ValueError as e: if 'does not support `chat`' in str(e): raise SystemExit(f'{model_path} is a scorer; use get_scores') from e else: raise

When it happens

Trigger: ChatModel(...) built with a reward/model classifier checkpoint (e.g. a merged reward model dir) and then calling .chat() or hitting /v1/chat/completions; loading an encoder-only or seq-cls model where AutoModelForCausalLM is not applicable.

Common situations: Reusing a fine-tuned reward-model output dir as a chat model; pointing model_name_or_path at a classifier; exporting then chatting with an RM checkpoint.

Related errors

• The current model does not support `stream_chat`.
• Cannot get scores using an auto-regressive model.
• vLLM not install, you may need to run `pip install vllm` or
• SGLang not install, you may need to run `pip install sglang[
• SGLang server initialization failed: {str(e)}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/71a650d486ad722e. Report an issue: GitHub.