huggingface/transformers · error · ValueError

ZeRO inference only makes sense with ZeRO Stage 3 - please a

Error message

ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config

What it means

Error "ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config" thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepspeed.py:620

    """
    from deepspeed.utils import logger as ds_logger

    model = trainer.model
    args = trainer.args

    hf_deepspeed_config = trainer.accelerator.state.deepspeed_plugin.hf_ds_config

    # resume config update - some bits like `model` and `num_training_steps` only become available during train
    hf_deepspeed_config.trainer_config_finalize(args, model, num_training_steps)

    # set the Deepspeed log level consistent with the Trainer
    ds_logger.setLevel(args.get_process_log_level())

    if inference:
        # only Z3 makes sense for the inference
        if not hf_deepspeed_config.is_zero3():
            raise ValueError("ZeRO inference only makes sense with ZeRO Stage 3 - please adjust your config")

        # in case the training config is re-used for inference
        hf_deepspeed_config.del_config_sub_tree("optimizer")
        hf_deepspeed_config.del_config_sub_tree("lr_scheduler")
        optimizer, lr_scheduler = None, None
        model_parameters = None
    else:
        trainer.optimizer = None  # important for when deepspeed_init is used as re-init
        deepspeed_tp_size = hf_deepspeed_config.config.get("tensor_parallel", {}).get("autotp_size", 1)
        if deepspeed_tp_size > 1:
            import deepspeed

            model = deepspeed.tp_model_init(
                model=model,
                tp_size=deepspeed_tp_size,
                dtype=hf_deepspeed_config.dtype(),
                config=hf_deepspeed_config.config,
            )

View on GitHub (pinned to a597f97485)

Solutions

1. Set the DeepSpeed config to ZeRO Stage 3 for ZeRO inference.
2. Use a different inference path if not using ZeRO Stage 3.

When it happens

Trigger: Raised in DeepSpeed ZeRO inference setup when the config is not ZeRO Stage 3.

Common situations: Enabling ZeRO inference with a Stage 1/2 DeepSpeed configuration.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a496209b92c822ba. Report an issue: GitHub.