hiyouga/LlamaFactory · error · ValueError

`predict_with_generate` is incompatible with DeepSpeed ZeRO-

Error message

`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.

What it means

Raised in parser.py:487 when predict_with_generate is enabled and is_deepspeed_zero3_enabled(). Under ZeRO-3, model parameters are sharded and gathered lazily, which is incompatible with HF Trainer's generation-based prediction path (weights are not materialized when generate runs).

Source

Thrown at src/llamafactory/hparams/parser.py:487

        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:

View on GitHub (pinned to f28afaf635)

Solutions

Switch to a ZeRO-2 (or lower) DeepSpeed config for the prediction run
Or drop `predict_with_generate` and evaluate via a separate non-ZeRO-3 run / llamafactory-cli chat or export + inference

Example fix

# before (YAML)
deepspeed: examples/deepspeed/ds_z3_config.json
predict_with_generate: true

# after
deepspeed: examples/deepspeed/ds_z2_config.json
predict_with_generate: true

Defensive patterns

Strategy: validation

Validate before calling

import json
def is_zero3(ds_path):
    if not ds_path: return False
    return json.load(open(ds_path)).get("zero_optimization", {}).get("stage") == 3
if config.get("predict_with_generate") and is_zero3(config.get("deepspeed")):
    raise SystemExit("predict_with_generate needs ZeRO stage <= 2")

Prevention

Keep separate prediction and ZeRO-3 training configs
Inspect zero_optimization.stage before combining generation eval with deepspeed

When it happens

Trigger: A config combining `predict_with_generate: true` with a DeepSpeed config whose zero_optimization.stage is 3 (e.g. ds_z3_config.json).

Common situations: Reusing a ZeRO-3 training YAML for an eval/prediction run that computes BLEU/ROUGE via generation; enabling ZeRO-3 to fit a large model and then turning on generation-based eval in the same run.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/22b388e1e8c9dbb3. Report an issue: GitHub.