{"record":{"id":"a3d333c0b447f411","repo":"hiyouga/LlamaFactory","slug":"vllm-engine-only-supports-auto-regressive-models","errorCode":null,"errorMessage":"vLLM engine only supports auto-regressive models.","messagePattern":"vLLM engine only supports auto-regressive models\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":678,"sourceCode":"        model_args.apply_kt_config(\n            finetuning_args,\n            training_args,\n            _get_kt_runtime_capacity(data_args, training_args, finetuning_args),\n        )\n\n    return model_args, data_args, training_args, finetuning_args, generating_args\n\n\ndef get_infer_args(args: dict[str, Any] | list[str] | None = None) -> _INFER_CLS:\n    model_args, data_args, finetuning_args, generating_args = _parse_infer_args(args)\n\n    # Setup logging\n    _set_transformers_logging()\n\n    # Check arguments\n    if model_args.infer_backend == \"vllm\":\n        if finetuning_args.stage != \"sft\":\n            raise ValueError(\"vLLM engine only supports auto-regressive models.\")\n\n        if model_args.quantization_bit is not None:\n            raise ValueError(\"vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).\")\n\n        if model_args.rope_scaling is not None:\n            raise ValueError(\"vLLM engine does not support RoPE scaling.\")\n\n        if model_args.adapter_name_or_path is not None and len(model_args.adapter_name_or_path) != 1:\n            raise ValueError(\"vLLM only accepts a single adapter. Merge them first.\")\n\n    _set_env_vars()\n    _verify_model_args(model_args, data_args, finetuning_args)\n    _check_extra_dependencies(model_args, finetuning_args)\n\n    # Post-process model arguments\n    if model_args.export_dir is not None and model_args.export_device == \"cpu\":\n        model_args.device_map = {\"\": torch.device(\"cpu\")}\n        if data_args.cutoff_len != DataArguments().cutoff_len:  # override cutoff_len if it is not default","sourceCodeStart":660,"sourceCodeEnd":696,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L660-L696","documentation":"Raised in get_infer_args when infer_backend is vllm but finetuning_args.stage is not 'sft'. The vLLM engine in LlamaFactory only serves plain causal language models; non-auto-regressive stages (e.g. rm for reward models) have no vLLM inference path.","triggerScenarios":"Calling chat/export with infer_backend: vllm while the stage field in the config is rm, ppo, kto, dpo or pt instead of sft.","commonSituations":"Reusing a chat YAML derived from a reward-model or DPO config and flipping only the backend to vllm; scripts that template stage across multiple configs.","solutions":["Set stage: sft (or remove a non-default stage) in the finetuning section when using infer_backend: vllm.","Switch infer_backend to hf if you genuinely need to run a non-auto-regressive model (e.g. a reward model)."],"exampleFix":"# before\ninfer_backend: vllm\nstage: rm\n\n# after\ninfer_backend: vllm\nstage: sft","handlingStrategy":"validation","validationCode":"if cfg[\"model_args\"][\"infer_backend\"] == \"vllm\":\n    assert cfg.get(\"finetuning_args\", {}).get(\"stage\", \"sft\") == \"sft\", \\\n        \"vLLM inference requires stage: sft\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive infer config from a known-good sft chat template instead of editing a training/rm config.","Lint stage and infer_backend together in a preflight check."],"tags":["vllm","infer-backend","stage","reward-model"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}