hiyouga/LlamaFactory · error · ValueError

Please use scripts/pissa_init.py to initialize PiSSA in Deep

Error message

Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.

What it means

Raised in parser.py:496 when finetuning_args.pissa_init is true and ZeRO-3 is enabled. PiSSA initialization factorizes the LoRA target matrices in-place, which requires full (unsharded) weight access; under ZeRO-3 the weights are partitioned so the init script must run beforehand.

Source

Thrown at src/llamafactory/hparams/parser.py:496

        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")

View on GitHub (pinned to f28afaf635)

Solutions

First run scripts/pissa_init.py to pre-compute and save the PiSSA-initialized base model, then point `model_name_or_path` at that output and disable pissa_init for the ZeRO-3 training run
Or train without ZeRO-3 (stage <= 2) so pissa_init can run inline

Example fix

# before (YAML)
deepspeed: examples/deepspeed/ds_z3_config.json
pissa_init: true

# after
# step 1: python scripts/pissa_init.py -m <model> -a <adapter> ...
# step 2 (YAML):
deepspeed: examples/deepspeed/ds_z3_config.json
model_name_or_path: <pissa_output_dir>
# pissa_init removed

Defensive patterns

Strategy: validation

Validate before calling

if config.get("pissa_init") and is_zero3(config.get("deepspeed")):
    raise SystemExit("Run scripts/pissa_init.py first, then train the converted model without pissa_init")

Prevention

Two-step PiSSA workflow: convert with the script, then train from its output
Never combine pissa_init with ZeRO-3 configs

When it happens

Trigger: Config combining `pissa_init: true` with a DeepSpeed ZeRO-3 config (stage 3).

Common situations: Enabling PiSSA (a fast-convergence LoRA variant) on a multi-GPU setup that already uses ZeRO-3 to fit the base model.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a5bca08713ba6b67. Report an issue: GitHub.