hiyouga/LlamaFactory · error · ValueError

vLLM engine does not support RoPE scaling.

Error message

vLLM engine does not support RoPE scaling.

What it means

Raised in get_infer_args when infer_backend is vllm and model_args.rope_scaling is not None. The vLLM engine in this codebase does not thread transformers-style rope_scaling dictionaries into its engine options, so the setting is rejected up front rather than silently ignored.

Source

Thrown at src/llamafactory/hparams/parser.py:684

    return model_args, data_args, training_args, finetuning_args, generating_args


def get_infer_args(args: dict[str, Any] | list[str] | None = None) -> _INFER_CLS:
    model_args, data_args, finetuning_args, generating_args = _parse_infer_args(args)

    # Setup logging
    _set_transformers_logging()

    # Check arguments
    if model_args.infer_backend == "vllm":
        if finetuning_args.stage != "sft":
            raise ValueError("vLLM engine only supports auto-regressive models.")

        if model_args.quantization_bit is not None:
            raise ValueError("vLLM engine does not support bnb quantization (GPTQ and AWQ are supported).")

        if model_args.rope_scaling is not None:
            raise ValueError("vLLM engine does not support RoPE scaling.")

        if model_args.adapter_name_or_path is not None and len(model_args.adapter_name_or_path) != 1:
            raise ValueError("vLLM only accepts a single adapter. Merge them first.")

    _set_env_vars()
    _verify_model_args(model_args, data_args, finetuning_args)
    _check_extra_dependencies(model_args, finetuning_args)

    # Post-process model arguments
    if model_args.export_dir is not None and model_args.export_device == "cpu":
        model_args.device_map = {"": torch.device("cpu")}
        if data_args.cutoff_len != DataArguments().cutoff_len:  # override cutoff_len if it is not default
            model_args.model_max_length = data_args.cutoff_len
    else:
        model_args.device_map = "auto"

    model_args.configure_kt_loading(finetuning_args, data_args.cutoff_len)

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove rope_scaling from the model args when using infer_backend: vllm.
2. Configure RoPE on the vLLM side instead via its own engine kwargs (e.g. rope_theta / long-context settings passed through vllm_extra_config if supported).
3. Use a model checkpoint whose rope scaling is already baked into its config, or switch to infer_backend: hf.

Example fix

# before
infer_backend: vllm
rope_scaling:
  rope_type: linear
  factor: 4.0

# after
infer_backend: vllm
# (omit rope_scaling; rely on model's native config)

if cfg["model_args"].get("infer_backend") == "vllm":
    assert not cfg["model_args"].get("rope_scaling"), \
        "rope_scaling is unsupported on the vLLM backend; remove it or use infer_backend: hf"

When it happens

Trigger: A chat config with infer_backend: vllm plus a rope_scaling: {...} block in the model section (common for long-context setups copied from HF configs).

Common situations: Copying the rope_scaling block from a training config or from a model card's long-context recipe into a vLLM chat config.

Related errors

• vLLM not install, you may need to run `pip install vllm` or
• vLLM engine does not support `get_scores`.
• `context_parallel_size` must be >= 1.
• KTransformers inference requires `infer_backend: huggingface
• PPO training is incompatible with S^2-Attn.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/47a1db0070b07590. Report an issue: GitHub.