{"record":{"id":"22b388e1e8c9dbb3","repo":"hiyouga/LlamaFactory","slug":"predict-with-generate-is-incompatible-with-deeps","errorCode":null,"errorMessage":"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.","messagePattern":"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":487,"sourceCode":"        raise ValueError(\"Please launch distributed training with `llamafactory-cli` or `torchrun`.\")\n\n    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:\n        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n\n    if training_args.max_steps == -1 and data_args.streaming:\n        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n\n    if training_args.do_train and data_args.dataset is None:\n        raise ValueError(\"Please specify dataset for training.\")\n\n    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (\n        data_args.eval_dataset is None and data_args.val_size < 1e-6\n    ):\n        raise ValueError(\"Please make sure eval_dataset be provided or val_size >1e-6\")\n\n    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L469-L505","documentation":"Raised in parser.py:487 when predict_with_generate is enabled and is_deepspeed_zero3_enabled(). Under ZeRO-3, model parameters are sharded and gathered lazily, which is incompatible with HF Trainer's generation-based prediction path (weights are not materialized when generate runs).","triggerScenarios":"A config combining `predict_with_generate: true` with a DeepSpeed config whose zero_optimization.stage is 3 (e.g. ds_z3_config.json).","commonSituations":"Reusing a ZeRO-3 training YAML for an eval/prediction run that computes BLEU/ROUGE via generation; enabling ZeRO-3 to fit a large model and then turning on generation-based eval in the same run.","solutions":["Switch to a ZeRO-2 (or lower) DeepSpeed config for the prediction run","Or drop `predict_with_generate` and evaluate via a separate non-ZeRO-3 run / llamafactory-cli chat or export + inference"],"exampleFix":"# before (YAML)\ndeepspeed: examples/deepspeed/ds_z3_config.json\npredict_with_generate: true\n\n# after\ndeepspeed: examples/deepspeed/ds_z2_config.json\npredict_with_generate: true","handlingStrategy":"validation","validationCode":"import json\ndef is_zero3(ds_path):\n    if not ds_path: return False\n    return json.load(open(ds_path)).get(\"zero_optimization\", {}).get(\"stage\") == 3\nif config.get(\"predict_with_generate\") and is_zero3(config.get(\"deepspeed\")):\n    raise SystemExit(\"predict_with_generate needs ZeRO stage <= 2\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep separate prediction and ZeRO-3 training configs","Inspect zero_optimization.stage before combining generation eval with deepspeed"],"tags":["deepspeed","zero-3","predict-with-generate","evaluation","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}