hiyouga/LlamaFactory · error · ValueError

Cannot use `predict_with_generate` and `compute_accuracy` to

Error message

Cannot use `predict_with_generate` and `compute_accuracy` together.

What it means

Raised in parser.py:490 when both predict_with_generate and finetuning_args.compute_accuracy are set. The two evaluation modes are mutually exclusive: generation-based metrics (BLEU/ROUGE) and logit-based accuracy cannot both be computed in one prediction pass.

Source

Thrown at src/llamafactory/hparams/parser.py:490

        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Decide the eval mode: keep compute_accuracy and remove predict_with_generate (logit accuracy)
2. Or keep predict_with_generate and remove compute_accuracy (generation metrics)
3. Run two separate configs if both kinds of metrics are needed

Example fix

# before (YAML)
predict_with_generate: true
compute_accuracy: true

# after
predict_with_generate: true
# compute_accuracy removed

if config.get("predict_with_generate") and config.get("compute_accuracy"):
    raise SystemExit("Choose one eval mode: predict_with_generate OR compute_accuracy")

When it happens

Trigger: YAML containing both `predict_with_generate: true` and `compute_accuracy: true` (compute_accuracy is typically set for RM or GTM/binary-classification style evals).

Common situations: Merging flags from two different example configs (a generation-eval example and an accuracy-eval example); enabling 'all eval options' when tuning a reward model.

Related errors

• Please make sure eval_dataset be provided or val_size >1e-6
• `predict_with_generate` is incompatible with DeepSpeed ZeRO-
• `save_dir` already exists, use another one.
• `virtual_pipeline_model_parallel_size` must be >= 1 when set
• `sequence_parallel` requires `tensor_model_parallel_size` >

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/718448e6888c28c9. Report an issue: GitHub.