hiyouga/LlamaFactory · error · ImportError

vLLM not install, you may need to run `pip install vllm` or

Error message

vLLM not install, you may need to run `pip install vllm`
or try to use HuggingFace backend: --infer_backend huggingface

What it means

ImportError raised by ChatModel.__init__ when infer_backend is vllm but importing VllmEngine fails. vLLM is an optional dependency; the chained original ImportError is preserved. The message tells you to either install vllm or fall back to the HuggingFace backend.

Source

Thrown at src/llamafactory/chat/chat_model.py:60

    Supports both sync and async methods.
    Sync methods: chat(), stream_chat() and get_scores().
    Async methods: achat(), astream_chat() and aget_scores().
    """

    def __init__(self, args: Optional[dict[str, Any]] = None) -> None:
        model_args, data_args, finetuning_args, generating_args = get_infer_args(args)

        if model_args.infer_backend == EngineName.HF:
            from .hf_engine import HuggingfaceEngine

            self.engine: BaseEngine = HuggingfaceEngine(model_args, data_args, finetuning_args, generating_args)
        elif model_args.infer_backend == EngineName.VLLM:
            try:
                from .vllm_engine import VllmEngine

                self.engine: BaseEngine = VllmEngine(model_args, data_args, finetuning_args, generating_args)
            except ImportError as e:
                raise ImportError(
                    "vLLM not install, you may need to run `pip install vllm`\n"
                    "or try to use HuggingFace backend: --infer_backend huggingface"
                ) from e
        elif model_args.infer_backend == EngineName.SGLANG:
            try:
                from .sglang_engine import SGLangEngine

                self.engine: BaseEngine = SGLangEngine(model_args, data_args, finetuning_args, generating_args)
            except ImportError as e:
                raise ImportError(
                    "SGLang not install, you may need to run `pip install sglang[all]`\n"
                    "or try to use HuggingFace backend: --infer_backend huggingface"
                ) from e
        else:
            raise NotImplementedError(f"Unknown backend: {model_args.infer_backend}")

        self._loop = asyncio.new_event_loop()
        self._thread = Thread(target=_start_background_loop, args=(self._loop,), daemon=True)

View on GitHub (pinned to f28afaf635)

Solutions

1. Install vllm in the same environment/interpreter (pip install vllm, ideally with the version pinned by this repo's requirements).
2. Or pass --infer_backend huggingface (infer_backend='huggingface') to use the HF engine.
3. Diagnose the chained cause: `python -c "from llamafactory.chat.vllm_engine import VllmEngine"` to see the underlying ImportError.
4. Verify torch/CUDA versions match vllm's requirements if the import fails despite installation.

Example fix

# before
ChatModel({'model_name_or_path': ..., 'infer_backend': 'vllm'})
# after (option 1)
pip install vllm
# after (option 2)
ChatModel({'model_name_or_path': ..., 'infer_backend': 'huggingface'})

def vllm_importable():
    try:
        import vllm  # noqa: F401
        return True
    except ImportError:
        return False

backend = "vllm" if vllm_importable() else "huggingface"

try { model = ChatModel({..., 'infer_backend': 'vllm'}) } except ImportError as e: if 'vLLM not install' in str(e): model = ChatModel({..., 'infer_backend': 'huggingface'}) else: raise

When it happens

Trigger: ChatModel(...) or llamafactory-cli chat/api with --infer_backend vllm on an environment where vllm is absent, partially installed (CUDA mismatch), or fails to import due to a broken dependency chain.

Common situations: Installing llamafactory without the [vllm] extra; vllm/torch version conflicts after an upgrade; running on CPU-only machines where vllm cannot import; container images trimmed of GPU libs.

Related errors

• SGLang not install, you may need to run `pip install sglang[
• vLLM engine does not support `get_scores`.
• KTransformers inference requires `infer_backend: huggingface
• Qwen2VL requires 3D position ids for mrope.
• Stage does not supported: {stage}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/64439dad9372dc88. Report an issue: GitHub.