hiyouga/LlamaFactory · error · ValueError

Cannot initialize PiSSA adapter on quantized models.

Error message

Cannot initialize PiSSA adapter on quantized models.

What it means

Raised in setup_adapter when training a quantized model with pissa_init enabled. PiSSA initialization decomposes and rewrites the base weight matrices via SVD, which cannot be done on quantized (bnb/GPTQ/AWQ) weights, so only standard LoRA init is allowed there.

Source

Thrown at src/llamafactory/model/adapter.py:335

def init_adapter(
    config: "PretrainedConfig",
    model: "PreTrainedModel",
    model_args: "ModelArguments",
    finetuning_args: "FinetuningArguments",
    is_trainable: bool,
) -> "PreTrainedModel":
    r"""Initialize the adapters.

    Support full-parameter, freeze and LoRA training.

    Note that the trainable parameters must be cast to float32.
    """
    if is_trainable and getattr(model, "quantization_method", None) is not None:
        if finetuning_args.finetuning_type not in ["lora", "oft"]:
            raise ValueError("Quantized models can only be used for the LoRA or OFT tuning.")

        if finetuning_args.pissa_init:
            raise ValueError("Cannot initialize PiSSA adapter on quantized models.")

    # cast trainable parameters to float32 if:
    # 1. is_trainable and not pure_bf16 and not badam and quantization_bit is not None (qlora)
    # 2. is_trainable and not pure_bf16 and not badam and not zero3 (zero3 already in fp32)
    cast_trainable_params_to_fp32 = False
    if not is_trainable:
        pass
    elif finetuning_args.pure_bf16 or finetuning_args.use_badam:
        logger.info_rank0("Pure bf16 / BAdam detected, remaining trainable params in half precision.")
    elif model_args.quantization_bit is None and is_deepspeed_zero3_enabled():
        logger.info_rank0("DeepSpeed ZeRO3 detected, remaining trainable params in float32.")
    else:
        logger.info_rank0("Upcasting trainable params to float32.")
        cast_trainable_params_to_fp32 = True

    if finetuning_args.finetuning_type == "full":
        _setup_full_tuning(model, finetuning_args, is_trainable, cast_trainable_params_to_fp32)
    elif finetuning_args.finetuning_type == "freeze":

View on GitHub (pinned to f28afaf635)

Solutions

1. Set pissa_init: false on quantized models (plain QLoRA init).
2. If PiSSA matters more than memory, train unquantized with pissa_init: true.

Example fix

# before
quantization_bit: 4
pissa_init: true

# after
quantization_bit: 4
pissa_init: false

if cfg["finetuning_args"].get("pissa_init"):
    assert cfg["model_args"].get("quantization_bit") is None, \
        "PiSSA init requires unquantized weights; disable pissa_init or quantization_bit"

When it happens

Trigger: quantization_bit set (or PTQ checkpoint) plus pissa_init: true and finetuning_type: lora.

Common situations: Copying a PiSSA experiment config onto a QLoRA setup hoping to combine both benefits.

Related errors

• Quantization is only compatible with the LoRA or OFT method.
• Please use scripts/pissa_init.py to initialize PiSSA for a q
• Cannot use device map for quantized models in training.
• Quantized models can only be used for the LoRA or OFT tuning
• Only 4-bit quantized model can use fsdp+qlora or auto device

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/4cf2fbe61b5352b4. Report an issue: GitHub.