{"record":{"id":"261f92a22bd2c239","repo":"Lightning-AI/pytorch-lightning","slug":"deepspeed-was-unable-to-load-the-checkpoint-ensur-261f92","errorCode":null,"errorMessage":"DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file with `Trainer(strategy=DeepSpeedStrategy(load_full_weights=True))`.","messagePattern":"DeepSpeed was unable to load the checkpoint\\. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file with `Trainer\\(strategy=DeepSpeedStrategy\\(load_full_weights=True\\)\\)`\\.","errorType":"exception","errorClass":"MisconfigurationException","httpStatus":null,"severity":"critical","filePath":"src/lightning/pytorch/strategies/deepspeed.py","lineNumber":700,"sourceCode":"            return super().load_checkpoint(checkpoint_path, weights_only)\n\n        _validate_checkpoint_directory(checkpoint_path)\n\n        # Rely on deepspeed to load the checkpoint and necessary information\n        assert self.lightning_module is not None\n\n        from lightning.pytorch.trainer.states import TrainerFn\n\n        is_fitting = self.lightning_module.trainer.state.fn == TrainerFn.FITTING\n\n        _, client_state = self.deepspeed_engine.load_checkpoint(\n            checkpoint_path,\n            load_optimizer_states=is_fitting,\n            load_lr_scheduler_states=False,\n            load_module_strict=self.lightning_module.strict_loading,\n        )\n        if client_state is None:\n            raise MisconfigurationException(\n                \"DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint \"\n                \"or a single checkpoint file with `Trainer(strategy=DeepSpeedStrategy(load_full_weights=True))`.\"\n            )\n        return client_state\n\n    @property\n    @override\n    def lightning_restore_optimizer(self) -> bool:\n        assert self.lightning_module is not None\n        # managed by DeepSpeed\n        if self.load_full_weights and self.zero_stage_3 and self.lightning_module.trainer.state.fn == TrainerFn.FITTING:\n            rank_zero_warn(\n                \"A single checkpoint file has been given. This means optimizer states cannot be restored.\"\n                \" If you'd like to restore these states, you must provide a path to the originally saved DeepSpeed\"\n                \" checkpoint. When using ZeRO 3, the original path should be a directory.\"\n            )\n        return False\n","sourceCodeStart":682,"sourceCodeEnd":718,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/deepspeed.py#L682-L718","documentation":"After DeepSpeed's `load_checkpoint` returns, a None client_state means DeepSpeed did not recognize the path as a checkpoint it created (its engine returns the user-state dict only for its own tag-based checkpoints). Lightning surfaces this as MisconfigurationException telling you to either pass a DeepSpeed checkpoint or enable load_full_weights to read a consolidated file.","triggerScenarios":"Calling `trainer.strategy.load_checkpoint(path)` / `trainer.fit(..., ckpt_path=...)` with DeepSpeedStrategy pointing at (a) a plain Lightning `.ckpt` file while `load_full_weights=False`, (b) a directory missing DeepSpeed's tags (zero-to_fp32.py, latest etc.), or (c) a corrupt/foreign checkpoint dir.","commonSituations":"Resuming from a checkpoint saved by a different strategy (DDP/FSDP) without `DeepSpeedStrategy(load_full_weights=True)`; copying only the .ckpt file out of a sharded checkpoint dir; path typos resolving to an existing but wrong dir.","solutions":["If resuming from a full single-file Lightning checkpoint, construct `DeepSpeedStrategy(load_full_weights=True)`","If resuming a ZeRO sharded run, point `ckpt_path` at the DeepSpeed checkpoint directory (containing the tags), not just the .ckpt file","Verify the checkpoint was produced by DeepSpeedStrategy (look for `zero/` subfolder, `latest` tag file) before resuming"],"exampleFix":"# before\nstrategy = DeepSpeedStrategy(config=cfg)\ntrainer = Trainer(strategy=strategy)\ntrainer.fit(model, ckpt_path=\"last.ckpt\")  # single-file ckpt\n\n# after\nstrategy = DeepSpeedStrategy(config=cfg, load_full_weights=True)\ntrainer = Trainer(strategy=strategy)\ntrainer.fit(model, ckpt_path=\"last.ckpt\")","handlingStrategy":"try-catch","validationCode":"from pathlib import Path\ndef is_deepspeed_ckpt(p):\n    p = Path(p)\n    return p.is_dir() and (p / \"latest\").exists()\n# choose load mode before resume\nfull = not is_deepspeed_ckpt(ckpt_path)","typeGuard":"def is_deepspeed_checkpoint(path: str) -> bool:\n    p = Path(path)\n    return p.is_dir() and (p / \"latest\").exists()","tryCatchPattern":"from lightning.pytorch.utilities.exceptions import MisconfigurationException\ntry:\n    trainer.fit(model, ckpt_path=ckpt)\nexcept MisconfigurationException as e:\n    if \"DeepSpeed was unable to load\" in str(e):\n        trainer.strategy = DeepSpeedStrategy(config=cfg, load_full_weights=True)\n        raise SystemExit(\"Retry with load_full_weights=True\")\n    raise","preventionTips":["Always pass the full DeepSpeed checkpoint directory (with tags) for sharded resumes","Use load_full_weights=True when the checkpoint came from another strategy","Keep checkpoint dirs intact — copy whole directories, not single .ckpt files"],"tags":["deepspeed","load-checkpoint","resume","load-full-weights"],"backgroundTag":"checkpoint-format-mismatch","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}