hiyouga/LlamaFactory · error · ValueError

KTransformers inference requires `infer_backend: huggingface

Error message

KTransformers inference requires `infer_backend: huggingface`.

What it means

Raised by configure_kt_loading when use_kt is true and infer_backend is anything other than EngineName.HF. KT's inference path hooks into transformers' from_pretrained via transformers.integrations.kt.configure_kt; vLLM/SGLang engines bypass that hook entirely, so the combination is rejected up front.

Source

Thrown at src/llamafactory/hparams/model_args.py:686

            self._get_advanced_kt_config(training_args),
        )
        update_kt_config = getattr(training_args, "update_kt_config", None)
        if not callable(update_kt_config):
            raise RuntimeError(
                "The installed Transformers-KT does not provide `TrainingArguments.update_kt_config()`."
            )

        adapter_dir = self._resolve_kt_adapter_artifact_dir("training")
        update_kt_config(kt_config, adapter_name_or_path=adapter_dir)

    def configure_kt_loading(self, finetuning_args: Any, model_max_length: int | None) -> None:
        r"""Configure KT model loading for inference and evaluation."""
        if not self.use_kt:
            if self._kt_inference_config is not None:
                raise ValueError("`kt_config` requires `use_kt: true`.")
            return
        if self.infer_backend != EngineName.HF:
            raise ValueError("KTransformers inference requires `infer_backend: huggingface`.")

        adapter_dir = self._resolve_kt_adapter_artifact_dir("inference")

        try:
            from transformers.integrations.kt import configure_kt
        except (ImportError, ModuleNotFoundError) as exc:
            raise RuntimeError("The installed Transformers-KT does not provide `configure_kt()`.") from exc

        kt_config = self.get_kt_config_dict(
            finetuning_args,
            model_max_length,
            self._normalize_advanced_kt_config(self._kt_inference_config),
        )
        self._kt_adapter_artifact_path = adapter_dir
        self._kt_config_handle = configure_kt(kt_config)


@dataclass

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `infer_backend: huggingface` in the inference/eval YAML when using KTransformers.
2. Or export/merge the KT LoRA adapter into a standard checkpoint (`llamafactory-cli export`) and serve that with vLLM/SGLang without use_kt.

Example fix

# before (yaml)
use_kt: true
infer_backend: vllm

# after (yaml)
use_kt: true
infer_backend: huggingface

if cfg.get('use_kt') and cfg.get('infer_backend', 'huggingface') != 'huggingface':
    raise SystemExit('KT inference requires infer_backend: huggingface')

When it happens

Trigger: A chat/eval/api YAML with use_kt: true plus infer_backend: vllm (or sglang); configure_kt_loading compares infer_backend to EngineName.HF and raises.

Common situations: Users train with KT LoRA, then reuse the config for vLLM-served inference only changing the stage, expecting the adapter to load transparently.

Related errors

• vLLM not install, you may need to run `pip install vllm` or
• SGLang not install, you may need to run `pip install sglang[
• SGLang server initialization failed: {str(e)}.
• SGLang only supports n=1.
• SGLang server error: {response.status_code}, {response.text}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/69260b55db37d0a9. Report an issue: GitHub.