hiyouga/LlamaFactory · error · ValueError

Cannot get scores using an auto-regressive model.

Error message

Cannot get scores using an auto-regressive model.

What it means

ValueError raised by HuggingfaceEngine.get_scores when self.can_generate is true — i.e. the loaded model is an auto-regressive (generative) causal LM, but scoring requires a sequence-classification/reward model. The engine enforces the inverse constraint of chat/stream_chat.

Source

Thrown at src/llamafactory/chat/hf_engine.py:419

            audios,
            input_kwargs,
        )
        async with self.semaphore:
            stream = self._stream_chat(*input_args)
            while True:
                try:
                    yield await asyncio.to_thread(stream)
                except StopAsyncIteration:
                    break

    @override
    async def get_scores(
        self,
        batch_input: list[str],
        **input_kwargs,
    ) -> list[float]:
        if self.can_generate:
            raise ValueError("Cannot get scores using an auto-regressive model.")

        input_args = (self.model, self.tokenizer, batch_input, input_kwargs)
        async with self.semaphore:
            return await asyncio.to_thread(self._get_scores, *input_args)

View on GitHub (pinned to f28afaf635)

Solutions

1. Point model_name_or_path at a reward/sequence-classification checkpoint (trained with stage rm).
2. Use chat/stream_chat if you actually want generations from a causal LM.
3. Check config.json: ForSequenceClassification architecture indicates a scorer; ForCausalLM indicates a generator.

Example fix

# before
chat_model = ChatModel({'model_name_or_path': 'Qwen/Qwen2.5-7B-Instruct'})
scores = await chat_model.aget_scores([...])
# after
chat_model = ChatModel({'model_name_or_path': 'outputs/rm_checkpoint'})
scores = await chat_model.aget_scores([...])

import json
def is_scorer(model_dir):
    archs = json.load(open(f"{model_dir}/config.json"))["architectures"]
    return any("SequenceClassification" in a for a in archs)

assert is_scorer(model_path)  # before calling get_scores

try { scores = await chat_model.aget_scores(batch) } except ValueError as e: if 'auto-regressive' in str(e): raise SystemExit(f'{model_path} cannot score; load an RM checkpoint') from e else: raise

When it happens

Trigger: Calling get_scores / the score-evaluation endpoint with a standard instruct LLM (e.g. Qwen2.5-Instruct) loaded on the HF backend; running reward-data evaluation against the base chat model instead of the trained RM.

Common situations: Forgetting to switch model_name_or_path to the trained reward model before scoring; assuming any model can produce log-prob-based scores via this API.

Related errors

• The current model does not support `chat`.
• The current model does not support `stream_chat`.
• Invalid request
• SGLang engine does not support `get_scores`.
• vLLM engine does not support `get_scores`.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e4a009a1bca32a1e. Report an issue: GitHub.