{"record":{"id":"32d5e7227f5cc7dd","repo":"Lightning-AI/pytorch-lightning","slug":"deepspeed-was-unable-to-load-the-checkpoint-ensur","errorCode":null,"errorMessage":"DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file by setting `DeepSpeedStrategy(..., load_full_weights=True)`.","messagePattern":"DeepSpeed was unable to load the checkpoint\\. Ensure you passed in a DeepSpeed compatible checkpoint or a single checkpoint file by setting `DeepSpeedStrategy\\(\\.\\.\\., load_full_weights=True\\)`\\.","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/lightning/fabric/strategies/deepspeed.py","lineNumber":543,"sourceCode":"                \" states, call the load method for each model checkpoint separately.\"\n            )\n        engine = engines[0]\n\n        from deepspeed.runtime.base_optimizer import DeepSpeedOptimizer\n\n        optimzer_state_requested = any(isinstance(item, (Optimizer, DeepSpeedOptimizer)) for item in state.values())\n\n        torch.cuda.empty_cache()\n        _, client_state = engine.load_checkpoint(\n            path,\n            tag=\"checkpoint\",\n            load_optimizer_states=optimzer_state_requested,\n            load_lr_scheduler_states=False,\n            load_module_strict=strict,\n        )\n\n        if client_state is None:\n            raise RuntimeError(\n                \"DeepSpeed was unable to load the checkpoint. Ensure you passed in a DeepSpeed compatible checkpoint\"\n                \" or a single checkpoint file by setting `DeepSpeedStrategy(..., load_full_weights=True)`.\"\n            )\n\n        # `Engine.load_checkpoint` adds useless keys 'optimizer' and 'lr_scheduler' to the client state; remove\n        # them to avoid name collision with user state\n        keys = set(client_state) & set(state) - {\"optimizer\", \"lr_scheduler\"}\n        _move_state_into(source=client_state, destination=state, keys=keys)\n        return client_state\n\n    @override\n    def clip_gradients_norm(\n        self,\n        module: \"DeepSpeedEngine\",\n        optimizer: Optimizer,\n        max_norm: Union[float, int],\n        norm_type: Union[float, int] = 2.0,\n        error_if_nonfinite: bool = True,","sourceCodeStart":525,"sourceCodeEnd":561,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/deepspeed.py#L525-L561","documentation":"DeepSpeedStrategy.load_checkpoint calls DeepSpeed's Engine.load_checkpoint, which returns a client_state dict on success. If DeepSpeed returns None, the checkpoint at the given path was not a DeepSpeed-compatible checkpoint (no loadable checkpoint state was found), so Lightning raises RuntimeError. The usual cause is pointing at a single consolidated .ckpt/.pth file while the strategy is in sharded mode, or a corrupt/foreign directory.","triggerScenarios":"Calling fabric.load_checkpoint(...) or strategy.load_checkpoint(...) with a path that is not a DeepSpeed engine checkpoint (e.g. a plain PyTorch state_dict file, an FSDP checkpoint, or a non-checkpoint path) while using DeepSpeedStrategy without load_full_weights=True.","commonSituations":"User saved a checkpoint with another strategy or plain torch.save and tries to resume with DeepSpeed; passing the wrong directory (missing the global step subfolder); mixed Lightning version upgrade changing checkpoint layout; remote URI not accessible so DeepSpeed finds nothing.","solutions":["Verify the path is a DeepSpeed checkpoint: it should contain a checkpoint/ subfolder with zero_to_* files or a single-file full checkpoint","If you have a single consolidated checkpoint file, construct DeepSpeedStrategy(..., load_full_weights=True) so the whole file is loaded directly","Re-save the checkpoint from a DeepSpeed run before trying to resume","Check that the checkpoint was produced by the same code path (fabric.save_checkpoint) you are now loading with"],"exampleFix":"# before\nstrategy = DeepSpeedStrategy(config=ds_config)\nfabric.load_checkpoint(\"model.ckpt\", state)  # RuntimeError\n\n# after\nstrategy = DeepSpeedStrategy(config=ds_config, load_full_weights=True)\nfabric.load_checkpoint(\"model.ckpt\", state)","handlingStrategy":"validation","validationCode":"from lightning.fabric.strategies.deepspeed import _is_deepspeed_checkpoint\nfrom fsspec import filesystem\npath = \"ckpt/global_step10\"\nif not _is_deepspeed_checkpoint(path, filesystem(\"file\")):\n    if Path(path).is_file():\n        strategy = DeepSpeedStrategy(..., load_full_weights=True)  # single-file path\n    else:\n        raise ValueError(f\"{path} is not a DeepSpeed checkpoint\")","typeGuard":null,"tryCatchPattern":"try:\n    fabric.load_checkpoint(path, state)\nexcept RuntimeError as e:\n    if \"unable to load the checkpoint\" in str(e):\n        # fall back to full-weights load or re-save from a DeepSpeed run\n        ...","preventionTips":["Always save with the same strategy you load with","Keep the whole save directory (tag subfolders + checkpoint/) intact","Test checkpoint resumption in CI with a tiny run"],"tags":["deepspeed","checkpoint","load"],"backgroundTag":"checkpoint-load-failed","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}