Lightning-AI/pytorch-lightning · error · MisconfigurationException

Do not set `gradient_accumulation_steps` in the DeepSpeed co

Error message

Do not set `gradient_accumulation_steps` in the DeepSpeed config as this will be set with the `accumulate_grad_batches` argument passed via the Lightning Trainer.

What it means

The Trainer owns gradient accumulation and injects `gradient_accumulation_steps` into the DeepSpeed config from `Trainer(accumulate_grad_batches=...)`. A user-set value would conflict with the Trainer's, so _format_batch_size_and_grad_accum_config raises MisconfigurationException when the key is already present.

Source

Thrown at src/lightning/pytorch/strategies/deepspeed.py:924

                    "pin_memory": pin_memory,
                }
            cfg = {
                "zero_allow_untested_optimizer": zero_allow_untested_optimizer,
                "zero_optimization": zero_config,
                **cfg,
            }
        if logging_batch_size_per_gpu != "auto":
            cfg = {"train_micro_batch_size_per_gpu": logging_batch_size_per_gpu, **cfg}
        return cfg

    def _format_batch_size_and_grad_accum_config(self) -> None:
        # TODO: Using Fabric, we do not support these variables within the config
        assert isinstance(self.config, dict)
        if self.lightning_module is None:
            return

        if "gradient_accumulation_steps" in self.config:
            raise MisconfigurationException(
                "Do not set `gradient_accumulation_steps` in the DeepSpeed config"
                " as this will be set with the `accumulate_grad_batches` argument passed via the Lightning Trainer."
            )
        self.config["gradient_accumulation_steps"] = self.lightning_module.trainer.accumulate_grad_batches
        if "train_micro_batch_size_per_gpu" not in self.config:
            batch_size = self._auto_select_batch_size()
            self.config["train_micro_batch_size_per_gpu"] = batch_size
        if "gradient_clipping" not in self.config:
            self.config["gradient_clipping"] = self.lightning_module.trainer.gradient_clip_val or 0.0

    def _auto_select_batch_size(self) -> int:
        # train_micro_batch_size_per_gpu is used for throughput logging purposes
        # by default we try to use the batch size of the loader
        assert self.lightning_module is not None
        batch_size = 1
        data_source = self.lightning_module.trainer.fit_loop._data_source
        if data_source.is_defined():
            train_dataloader = data_source.dataloader()

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Remove `gradient_accumulation_steps` from the DeepSpeed config and set it via `Trainer(accumulate_grad_batches=N)`
2. Keep all accumulation control in Lightning to guarantee the engine and loop agree

Example fix

# before
cfg = {"gradient_accumulation_steps": 4, "zero_optimization": {"stage": 2}}
trainer = Trainer(strategy=DeepSpeedStrategy(config=cfg))

# after
cfg = {"zero_optimization": {"stage": 2}}
trainer = Trainer(strategy=DeepSpeedStrategy(config=cfg), accumulate_grad_batches=4)

ds_config.pop("gradient_accumulation_steps", None)  # Lightning owns this key
trainer = Trainer(strategy=DeepSpeedStrategy(config=ds_config), accumulate_grad_batches=4)

When it happens

Trigger: Passing a DeepSpeed config dict/JSON that contains `gradient_accumulation_steps`, then calling `trainer.fit()` with DeepSpeedStrategy.

Common situations: Copying a DeepSpeed JSON from a pure-DeepSpeed (HuggingFace-style) script into Lightning; hand-tuned configs found online that include accumulation settings.

Related errors

• The `{type(trainer.strategy).__name__}` does not support `ac
• You have set `accumulate_grad_batches` and are using the `Gr
• No models were set up for backward. Did you forget to call `
• When using multiple models + deepspeed, please provide the m
• The `{type(self._strategy).__name__}` requires the model and

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/e66fcc195d5b170d. Report an issue: GitHub.