hiyouga/LlamaFactory · error · ValueError

When `adapter_name_or_path` is provided for training, only a

Error message

When `adapter_name_or_path` is provided for training, only a single LoRA adapter is supported. Training will continue on the specified adapter. Please merge multiple adapters before starting a new LoRA adapter.

What it means

When resuming LoRA training with adapter_name_or_path, the PEFT plugin only supports exactly one adapter for continued training; PEFT cannot train on top of multiple simultaneously loaded adapters. If is_train is true and more than one adapter path is given, it raises ValueError telling you to merge adapters first.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/peft.py:129


def load_adapter(model: HFModel, adapter_name_or_path: list[str] | str, is_train: bool) -> HFModel:
    r"""Loads adapter(s) into the model.

    Determine adapter usage based on mode:
    - Training: Load the single adapter for continued training.
    - Inference: Merge all adapters to clean up the model.
    - Unmergeable: Keep the single adapter active without merging.
    """
    if not isinstance(adapter_name_or_path, list):
        adapter_name_or_path = [adapter_name_or_path]

    # TODO
    # Adapters fix for deepspeed and quant
    # Adapters fix for vision

    if is_train and len(adapter_name_or_path) > 1:
        raise ValueError(
            "When `adapter_name_or_path` is provided for training, only a single LoRA adapter is supported. "
            "Training will continue on the specified adapter. "
            "Please merge multiple adapters before starting a new LoRA adapter."
        )

    if is_train:
        adapter_to_merge = []
        adapter_to_resume = adapter_name_or_path[0]
    else:
        adapter_to_merge = adapter_name_or_path
        adapter_to_resume = None

    if adapter_to_merge:
        model = merge_adapters(model, adapter_to_merge)

    if adapter_to_resume is not None:
        model = PeftModel.from_pretrained(model, adapter_to_resume, is_trainable=is_train)
        if is_train:

View on GitHub (pinned to f28afaf635)

Solutions

Keep only the single adapter you want to continue training in adapter_name_or_path
Merge the other adapters first (llamafactory-cli export / merge_and_unload) and point training at the merged model, or chain adapters by training sequentially
For multi-adapter inference, ensure the stage is inference so is_train is false and merging applies

Example fix

# before
adapter_name_or_path:
  - saves/lora_v1
  - saves/lora_v2
stage: sft

# after
adapter_name_or_path: saves/lora_v2
stage: sft
# (or merge lora_v1 into the base model and train from the merged checkpoint)

Defensive patterns

Strategy: validation

Validate before calling

if stage in {"sft","dpo","rm","kto","pt"} and isinstance(adapter_name_or_path, list) and len(adapter_name_or_path) > 1:
    raise ValueError("training supports a single adapter; merge extras first")

Type guard

def is_single_adapter_for_training(paths, is_train: bool) -> bool:
    return not is_train or (isinstance(paths, str) or len(paths) == 1)

Prevention

Use a single adapter path in train configs; reserve multi-adapter lists for inference
Merge finished adapters (export/merge_and_unload) before stacking new LoRA training

When it happens

Trigger: Passing adapter_name_or_path as a list with 2+ entries (e.g. ["lora/v1","lora/v2"]) together with a training stage (is_train). Multi-adapter lists are only valid for inference, where they get merged.

Common situations: Copying an inference-style config (multiple adapters for merging/evaluation) into a train config; attempting to stack a new LoRA on several previous adapters; YAML lists under adapter_name_or_path for continued fine-tuning.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/21765114ea8d82ca. Report an issue: GitHub.