{"record":{"id":"1013fe10bcfac077","repo":"Lightning-AI/pytorch-lightning","slug":"when-saving-the-deepspeed-stage-3-checkpoint-each","errorCode":null,"errorMessage":"When saving the DeepSpeed Stage 3 checkpoint, each worker will save a shard of the checkpoint within a directory. If a single file is required after training, see https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed-zero-stage-3-single-file for instructions.","messagePattern":"When saving the DeepSpeed Stage 3 checkpoint, each worker will save a shard of the checkpoint within a directory\\. If a single file is required after training, see https://lightning\\.ai/docs/pytorch/stable/advanced/model_parallel\\.html#deepspeed-zero-stage-3-single-file for instructions\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/lightning/pytorch/strategies/deepspeed.py","lineNumber":658,"sourceCode":"            filepath: write-target file's path\n            storage_options: not used for ``DeepSpeedStrategy`` as ``CheckpointIO`` is not used\n\n        Raises:\n            TypeError:\n                If ``storage_options`` arg is passed in\n\n        \"\"\"\n        # broadcast the filepath from rank 0 to ensure all the states are saved in a common filepath\n        filepath = self.broadcast(filepath)\n\n        if storage_options is not None:\n            raise TypeError(\n                \"`Trainer.save_checkpoint(..., storage_options=...)` with `storage_options` arg\"\n                f\" is not supported for `{self.__class__.__name__}` as `CheckpointIO` is not used.\"\n            )\n\n        if self.zero_stage_3 and self._multi_device and self.is_global_zero:\n            warning_cache.warn(\n                \"When saving the DeepSpeed Stage 3 checkpoint, \"\n                \"each worker will save a shard of the checkpoint within a directory. \"\n                \"If a single file is required after training, \"\n                \"see https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#\"\n                \"deepspeed-zero-stage-3-single-file for instructions.\"\n            )\n        # Use deepspeed's internal checkpointing function to handle partitioned weights across processes\n        # dump states as a checkpoint dictionary object\n        _exclude_keys = [\"state_dict\", \"optimizer_states\"]\n        checkpoint = {k: v for k, v in checkpoint.items() if k not in _exclude_keys}\n        self.deepspeed_engine.save_checkpoint(\n            filepath,\n            client_state=checkpoint,\n            tag=\"checkpoint\",\n            exclude_frozen_parameters=self.exclude_frozen_parameters,\n        )\n\n    @override","sourceCodeStart":640,"sourceCodeEnd":676,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/deepspeed.py#L640-L676","documentation":"With DeepSpeed ZeRO Stage 3 and multiple devices, saving a checkpoint writes one sharded directory per worker, not a single consolidated .ckpt file. Lightning warns and links docs for consolidating to a single file.","triggerScenarios":"trainer.save_checkpoint(...) or ModelCheckpoint with Trainer(strategy=deepspeed_stage_3) on multi-GPU/multi-node.","commonSituations":"Expecting a portable single-file checkpoint for inference/serving after stage-3 training; resuming on different hardware.","solutions":["Follow the linked docs: load the sharded checkpoint with DeepSpeed's conversion, or use zero_to_fp32.py to consolidate weights into a single file","Resume training directly from the sharded directory (supported)","If single-file checkpoints are essential, use stage 2 or zero_offload variants without sharded state"],"exampleFix":"# before\ntrainer.save_checkpoint('model.ckpt')  # produces sharded dir with stage3 multi-device\n# after\n# after training, consolidate:\n# python zero_to_f32.py path/to/sharded_ckpt model.ckpt","handlingStrategy":"fallback","validationCode":"assert not (strategy.zero_stage_3 and trainer.num_devices > 1) or allow_shards, 'stage-3 multi-device saves sharded dir'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Plan a zero_to_fp32 consolidation step in stage-3 pipelines","Document that resume must point at the sharded directory"],"tags":["deepspeed","zero-stage-3","checkpoint","sharding"],"backgroundTag":"sharded-checkpoint-saved","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}