Lightning-AI/pytorch-lightning · error · TypeError

`FSDPStrategy.save_checkpoint(..., storage_options=...)` is

Error message

`FSDPStrategy.save_checkpoint(..., storage_options=...)` is not supported because `FSDPStrategy` does not use the `CheckpointIO`.

What it means

FSDPStrategy implements checkpoint saving itself (sharded dirs or consolidated files via torch.distributed.checkpoint) and does not route through the CheckpointIO plugin abstraction, so per-backend `storage_options` cannot be honored and passing them raises TypeError.

Source

Thrown at src/lightning/pytorch/strategies/fsdp.py:570

                state_dict = FSDP.optim_state_dict(self.model, optimizer)
                if self.global_rank == 0:
                    # Store the optimizer state dict in standard format
                    state_dict = FSDP.rekey_optim_state_dict(state_dict, OptimStateKeyType.PARAM_ID, self.model)
                return state_dict

        raise ValueError(f"Unknown state_dict_type: {self._state_dict_type}")

    @override
    def load_optimizer_state_dict(self, checkpoint: Mapping[str, Any]) -> None:
        # Override to do nothing, the FSDP already loaded the states in `load_checkpoint()`
        pass

    @override
    def save_checkpoint(
        self, checkpoint: dict[str, Any], filepath: _PATH, storage_options: Optional[Any] = None
    ) -> None:
        if storage_options is not None:
            raise TypeError(
                "`FSDPStrategy.save_checkpoint(..., storage_options=...)` is not supported because"
                " `FSDPStrategy` does not use the `CheckpointIO`."
            )

        path = _resolve_path(self.broadcast(filepath))
        if self._state_dict_type == "full" and _is_checkpoint_dir(path) and not _is_sharded_checkpoint(path):
            raise IsADirectoryError(f"The checkpoint path exists and is a directory: {path}")

        if self._state_dict_type == "sharded":
            _prepare_directory_checkpoint(path)

            converted_state = {"model": checkpoint.pop("state_dict")}
            converted_state.update({
                f"optimizer_{idx}": optim_state
                for idx, optim_state in enumerate(checkpoint.pop("optimizer_states", []))
            })

            _distributed_checkpoint_save(converted_state, path)

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Remove `storage_options` from the save_checkpoint call under FSDP
2. Handle remote upload after the local save completes
3. Switch strategy or checkpoint plugin if backend-specific options are required

Example fix

# before
trainer.save_checkpoint("ckpt", storage_options={"auto_mkdir": True})

# after
trainer.save_checkpoint("ckpt")

from lightning.pytorch.strategies import FSDPStrategy
if isinstance(trainer.strategy, FSDPStrategy):
    trainer.save_checkpoint(path)  # no storage_options
else:
    trainer.save_checkpoint(path, storage_options=opts)

def supports_storage_options(trainer) -> bool:
    return getattr(trainer.strategy, "checkpoint_io", None) is not None

When it happens

Trigger: `trainer.save_checkpoint(path, storage_options=...)` or `strategy.save_checkpoint(checkpoint, filepath, storage_options=...)` while training under FSDPStrategy.

Common situations: Shared checkpoint utility code that passes storage_options for fsspec/S3 backends; pipelines written for AsyncCheckpointIO reused with FSDP.

Related errors

• `FSDPStrategy.save_checkpoint(..., storage_options=...)` is
• `Trainer.save_checkpoint(..., storage_options=...)` with `st
• The checkpoint path exists and is a directory: {path}
• The optimizer has references to the model's meta-device para
• The optimizer has references to the model's meta-device para

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/284d243a9b091615. Report an issue: GitHub.