{"record":{"id":"4cf2fbe61b5352b4","repo":"hiyouga/LlamaFactory","slug":"cannot-initialize-pissa-adapter-on-quantized-model","errorCode":null,"errorMessage":"Cannot initialize PiSSA adapter on quantized models.","messagePattern":"Cannot initialize PiSSA adapter on quantized models\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/adapter.py","lineNumber":335,"sourceCode":"def init_adapter(\n    config: \"PretrainedConfig\",\n    model: \"PreTrainedModel\",\n    model_args: \"ModelArguments\",\n    finetuning_args: \"FinetuningArguments\",\n    is_trainable: bool,\n) -> \"PreTrainedModel\":\n    r\"\"\"Initialize the adapters.\n\n    Support full-parameter, freeze and LoRA training.\n\n    Note that the trainable parameters must be cast to float32.\n    \"\"\"\n    if is_trainable and getattr(model, \"quantization_method\", None) is not None:\n        if finetuning_args.finetuning_type not in [\"lora\", \"oft\"]:\n            raise ValueError(\"Quantized models can only be used for the LoRA or OFT tuning.\")\n\n        if finetuning_args.pissa_init:\n            raise ValueError(\"Cannot initialize PiSSA adapter on quantized models.\")\n\n    # cast trainable parameters to float32 if:\n    # 1. is_trainable and not pure_bf16 and not badam and quantization_bit is not None (qlora)\n    # 2. is_trainable and not pure_bf16 and not badam and not zero3 (zero3 already in fp32)\n    cast_trainable_params_to_fp32 = False\n    if not is_trainable:\n        pass\n    elif finetuning_args.pure_bf16 or finetuning_args.use_badam:\n        logger.info_rank0(\"Pure bf16 / BAdam detected, remaining trainable params in half precision.\")\n    elif model_args.quantization_bit is None and is_deepspeed_zero3_enabled():\n        logger.info_rank0(\"DeepSpeed ZeRO3 detected, remaining trainable params in float32.\")\n    else:\n        logger.info_rank0(\"Upcasting trainable params to float32.\")\n        cast_trainable_params_to_fp32 = True\n\n    if finetuning_args.finetuning_type == \"full\":\n        _setup_full_tuning(model, finetuning_args, is_trainable, cast_trainable_params_to_fp32)\n    elif finetuning_args.finetuning_type == \"freeze\":","sourceCodeStart":317,"sourceCodeEnd":353,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/adapter.py#L317-L353","documentation":"Raised in setup_adapter when training a quantized model with pissa_init enabled. PiSSA initialization decomposes and rewrites the base weight matrices via SVD, which cannot be done on quantized (bnb/GPTQ/AWQ) weights, so only standard LoRA init is allowed there.","triggerScenarios":"quantization_bit set (or PTQ checkpoint) plus pissa_init: true and finetuning_type: lora.","commonSituations":"Copying a PiSSA experiment config onto a QLoRA setup hoping to combine both benefits.","solutions":["Set pissa_init: false on quantized models (plain QLoRA init).","If PiSSA matters more than memory, train unquantized with pissa_init: true."],"exampleFix":"# before\nquantization_bit: 4\npissa_init: true\n\n# after\nquantization_bit: 4\npissa_init: false","handlingStrategy":"validation","validationCode":"if cfg[\"finetuning_args\"].get(\"pissa_init\"):\n    assert cfg[\"model_args\"].get(\"quantization_bit\") is None, \\\n        \"PiSSA init requires unquantized weights; disable pissa_init or quantization_bit\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat pissa_init and quantization_bit as mutually exclusive in config linting.","Choose between QLoRA memory savings and PiSSA init; they cannot be combined."],"tags":["pissa","qlora","lora-init","quantization"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}