{"record":{"id":"a496209b92c822ba","repo":"huggingface/transformers","slug":"zero-inference-only-makes-sense-with-zero-stage-3","errorCode":null,"errorMessage":"ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config","messagePattern":"ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepspeed.py","lineNumber":620,"sourceCode":"\n    \"\"\"\n    from deepspeed.utils import logger as ds_logger\n\n    model = trainer.model\n    args = trainer.args\n\n    hf_deepspeed_config = trainer.accelerator.state.deepspeed_plugin.hf_ds_config\n\n    # resume config update - some bits like `model` and `num_training_steps` only become available during train\n    hf_deepspeed_config.trainer_config_finalize(args, model, num_training_steps)\n\n    # set the Deepspeed log level consistent with the Trainer\n    ds_logger.setLevel(args.get_process_log_level())\n\n    if inference:\n        # only Z3 makes sense for the inference\n        if not hf_deepspeed_config.is_zero3():\n            raise ValueError(\"ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config\")\n\n        # in case the training config is re-used for inference\n        hf_deepspeed_config.del_config_sub_tree(\"optimizer\")\n        hf_deepspeed_config.del_config_sub_tree(\"lr_scheduler\")\n        optimizer, lr_scheduler = None, None\n        model_parameters = None\n    else:\n        trainer.optimizer = None  # important for when deepspeed_init is used as re-init\n        deepspeed_tp_size = hf_deepspeed_config.config.get(\"tensor_parallel\", {}).get(\"autotp_size\", 1)\n        if deepspeed_tp_size > 1:\n            import deepspeed\n\n            model = deepspeed.tp_model_init(\n                model=model,\n                tp_size=deepspeed_tp_size,\n                dtype=hf_deepspeed_config.dtype(),\n                config=hf_deepspeed_config.config,\n            )","sourceCodeStart":602,"sourceCodeEnd":638,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepspeed.py#L602-L638","documentation":"Error \"ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config\" thrown in huggingface/transformers.","triggerScenarios":"Raised in DeepSpeed ZeRO inference setup when the config is not ZeRO Stage 3.","commonSituations":"Enabling ZeRO inference with a Stage 1/2 DeepSpeed configuration.","solutions":["Set the DeepSpeed config to ZeRO Stage 3 for ZeRO inference.","Use a different inference path if not using ZeRO Stage 3."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}