hiyouga/LlamaFactory · error · NotImplementedError

vLLM engine does not support `get_scores`.

Error message

vLLM engine does not support `get_scores`.

What it means

vLLMEngine.get_scores unconditionally raises NotImplementedError. The vLLM chat backend in this codebase only exposes generation (chat/stream_chat); the sequence-scoring interface used for reward-model evaluation is not wired to vLLM's log-prob APIs, so any get_scores call fails fast.

Source

Thrown at src/llamafactory/chat/vllm_engine.py:273

        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> AsyncGenerator[str, None]:
        generated_text = ""
        generator = await self._generate(messages, system, tools, images, videos, audios, **input_kwargs)
        async for result in generator:
            delta_text = result.outputs[0].text[len(generated_text) :]
            generated_text = result.outputs[0].text
            yield delta_text

    @override
    async def get_scores(
        self,
        batch_input: list[str],
        **input_kwargs,
    ) -> list[float]:
        raise NotImplementedError("vLLM engine does not support `get_scores`.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Run reward scoring with the hf engine instead of vllm.
2. Guard call sites with an engine-capability check so scoring code never dispatches to vLLMEngine.

Example fix

# before
scores = await engine.get_scores(batch_input)  # engine is vLLMEngine

# after
if type(engine).__name__ == "vLLMEngine":
    raise SystemExit("scoring requires the hf engine")
scores = await engine.get_scores(batch_input)

assert type(engine).__name__ != "vLLMEngine", "vllm engine cannot score sequences"

def supports_scoring(engine) -> bool:
    return not hasattr(engine, "_generate") or type(engine).__name__ not in {"vLLMEngine", "SGLangEngine"}

When it happens

Trigger: Calling await engine.get_scores(batch_input) on a vLLMEngine instance — e.g. an RM evaluation or preference-scoring script whose engine backend was configured as vllm.

Common situations: Switching inference_backend to vllm for speed and forgetting that the scoring path still needs the hf engine; shared evaluation code that assumes every engine implements get_scores.

Related errors

• SGLang engine does not support `get_scores`.
• vLLM not install, you may need to run `pip install vllm` or
• SGLang only supports n=1.
• KTransformers inference requires `infer_backend: huggingface
• vLLM engine only supports auto-regressive models.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e29e82fddf388731. Report an issue: GitHub.