hiyouga/LlamaFactory · error · ValueError

When `adapter_name_or_path` is provided for training, only…

Error message

When `adapter_name_or_path` is provided for training, only a single LoRA adapter is supported. Training will continue on the specified adapter. Please merge multiple adapters before starting a new LoRA adapter.

What it means

When resuming LoRA training with adapter_name_or_path, the PEFT plugin only supports exactly one adapter for continued training; PEFT cannot train on top of multiple simultaneously loaded adapters. If is_train is true and more than one adapter path is given, it raises ValueError telling you to merge adapters first.

Solutions

  1. Keep only the single adapter you want to continue training in adapter_name_or_path
  2. Merge the other adapters first (llamafactory-cli export / merge_and_unload) and point training at the merged model, or chain adapters by training sequentially
  3. For multi-adapter inference, ensure the stage is inference so is_train is false and merging applies

Example fix

# before
adapter_name_or_path:
  - saves/lora_v1
  - saves/lora_v2
stage: sft

# after
adapter_name_or_path: saves/lora_v2
stage: sft
# (or merge lora_v1 into the base model and train from the merged checkpoint)
Defensive patterns

Strategy: validation

Validate before calling

if stage in {"sft","dpo","rm","kto","pt"} and isinstance(adapter_name_or_path, list) and len(adapter_name_or_path) > 1:
    raise ValueError("training supports a single adapter; merge extras first")

Type guard

def is_single_adapter_for_training(paths, is_train: bool) -> bool:
    return not is_train or (isinstance(paths, str) or len(paths) == 1)

Prevention

When it happens

Trigger: Passing adapter_name_or_path as a list with 2+ entries (e.g. ["lora/v1","lora/v2"]) together with a training stage (is_train). Multi-adapter lists are only valid for inference, where they get merged.

Common situations: Copying an inference-style config (multiple adapters for merging/evaluation) into a train config; attempting to stack a new LoRA on several previous adapters; YAML lists under adapter_name_or_path for continued fine-tuning.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/21765114ea8d82ca. Report an issue: GitHub.

Appendix: source

Thrown at src/llamafactory/v1/plugins/model_plugins/peft.py:129


def load_adapter(model: HFModel, adapter_name_or_path: list[str] | str, is_train: bool) -> HFModel:
    r"""Loads adapter(s) into the model.

    Determine adapter usage based on mode:
    - Training: Load the single adapter for continued training.
    - Inference: Merge all adapters to clean up the model.
    - Unmergeable: Keep the single adapter active without merging.
    """
    if not isinstance(adapter_name_or_path, list):
        adapter_name_or_path = [adapter_name_or_path]

    # TODO
    # Adapters fix for deepspeed and quant
    # Adapters fix for vision

    if is_train and len(adapter_name_or_path) > 1:
        raise ValueError(
            "When `adapter_name_or_path` is provided for training, only a single LoRA adapter is supported. "
            "Training will continue on the specified adapter. "
            "Please merge multiple adapters before starting a new LoRA adapter."
        )

    if is_train:
        adapter_to_merge = []
        adapter_to_resume = adapter_name_or_path[0]
    else:
        adapter_to_merge = adapter_name_or_path
        adapter_to_resume = None

    if adapter_to_merge:
        model = merge_adapters(model, adapter_to_merge)

    if adapter_to_resume is not None:
        model = PeftModel.from_pretrained(model, adapter_to_resume, is_trainable=is_train)
        if is_train:

View on GitHub (pinned to f28afaf635)