{"record":{"id":"718448e6888c28c9","repo":"hiyouga/LlamaFactory","slug":"cannot-use-predict-with-generate-and-compute-ac","errorCode":null,"errorMessage":"Cannot use `predict_with_generate` and `compute_accuracy` together.","messagePattern":"Cannot use `predict_with_generate` and `compute_accuracy` together\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":490,"sourceCode":"        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n\n    if training_args.max_steps == -1 and data_args.streaming:\n        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n\n    if training_args.do_train and data_args.dataset is None:\n        raise ValueError(\"Please specify dataset for training.\")\n\n    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (\n        data_args.eval_dataset is None and data_args.val_size < 1e-6\n    ):\n        raise ValueError(\"Please make sure eval_dataset be provided or val_size >1e-6\")\n\n    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n","sourceCodeStart":472,"sourceCodeEnd":508,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L472-L508","documentation":"Raised in parser.py:490 when both predict_with_generate and finetuning_args.compute_accuracy are set. The two evaluation modes are mutually exclusive: generation-based metrics (BLEU/ROUGE) and logit-based accuracy cannot both be computed in one prediction pass.","triggerScenarios":"YAML containing both `predict_with_generate: true` and `compute_accuracy: true` (compute_accuracy is typically set for RM or GTM/binary-classification style evals).","commonSituations":"Merging flags from two different example configs (a generation-eval example and an accuracy-eval example); enabling 'all eval options' when tuning a reward model.","solutions":["Decide the eval mode: keep compute_accuracy and remove predict_with_generate (logit accuracy)","Or keep predict_with_generate and remove compute_accuracy (generation metrics)","Run two separate configs if both kinds of metrics are needed"],"exampleFix":"# before (YAML)\npredict_with_generate: true\ncompute_accuracy: true\n\n# after\npredict_with_generate: true\n# compute_accuracy removed","handlingStrategy":"validation","validationCode":"if config.get(\"predict_with_generate\") and config.get(\"compute_accuracy\"):\n    raise SystemExit(\"Choose one eval mode: predict_with_generate OR compute_accuracy\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat the two eval flags as a radio button in config tooling","Document which stage each flag belongs to (compute_accuracy for RM/classification)"],"tags":["evaluation","mutually-exclusive","compute-accuracy","predict-with-generate","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}