hiyouga/LlamaFactory · error · ValueError

vLLM engine only supports auto-regressive models.

Error message

vLLM engine only supports auto-regressive models.

What it means

Raised in get_infer_args when infer_backend is vllm but finetuning_args.stage is not 'sft'. The vLLM engine in LlamaFactory only serves plain causal language models; non-auto-regressive stages (e.g. rm for reward models) have no vLLM inference path.

Source

Thrown at src/llamafactory/hparams/parser.py:678

        model_args.apply_kt_config(
            finetuning_args,
            training_args,
            _get_kt_runtime_capacity(data_args, training_args, finetuning_args),
        )

    return model_args, data_args, training_args, finetuning_args, generating_args


def get_infer_args(args: dict[str, Any] | list[str] | None = None) -> _INFER_CLS:
    model_args, data_args, finetuning_args, generating_args = _parse_infer_args(args)

    # Setup logging
    _set_transformers_logging()

    # Check arguments
    if model_args.infer_backend == "vllm":
        if finetuning_args.stage != "sft":
            raise ValueError("vLLM engine only supports auto-regressive models.")

        if model_args.quantization_bit is not None:
            raise ValueError("vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).")

        if model_args.rope_scaling is not None:
            raise ValueError("vLLM engine does not support RoPE scaling.")

        if model_args.adapter_name_or_path is not None and len(model_args.adapter_name_or_path) != 1:
            raise ValueError("vLLM only accepts a single adapter. Merge them first.")

    _set_env_vars()
    _verify_model_args(model_args, data_args, finetuning_args)
    _check_extra_dependencies(model_args, finetuning_args)

    # Post-process model arguments
    if model_args.export_dir is not None and model_args.export_device == "cpu":
        model_args.device_map = {"": torch.device("cpu")}
        if data_args.cutoff_len != DataArguments().cutoff_len:  # override cutoff_len if it is not default

View on GitHub (pinned to f28afaf635)

Solutions

1. Set stage: sft (or remove a non-default stage) in the finetuning section when using infer_backend: vllm.
2. Switch infer_backend to hf if you genuinely need to run a non-auto-regressive model (e.g. a reward model).

Example fix

# before
infer_backend: vllm
stage: rm

# after
infer_backend: vllm
stage: sft

if cfg["model_args"]["infer_backend"] == "vllm":
    assert cfg.get("finetuning_args", {}).get("stage", "sft") == "sft", \
        "vLLM inference requires stage: sft"

When it happens

Trigger: Calling chat/export with infer_backend: vllm while the stage field in the config is rm, ppo, kto, dpo or pt instead of sft.

Common situations: Reusing a chat YAML derived from a reward-model or DPO config and flipping only the backend to vllm; scripts that template stage across multiple configs.

Related errors

• vLLM engine does not support `get_scores`.
• vLLM/SGLang backend is only available for API, CLI and Web.
• vLLM not install, you may need to run `pip install vllm` or
• Cannot get scores using an auto-regressive model.
• SGLang engine does not support `get_scores`.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a3d333c0b447f411. Report an issue: GitHub.