{"record":{"id":"bccd255d89b36ea2","repo":"Lightning-AI/pytorch-lightning","slug":"directory-ds-checkpoint-dir-doesn-t-exist","errorCode":null,"errorMessage":"Directory '{ds_checkpoint_dir}' doesn't exist","messagePattern":"Directory '(.+?)' doesn't exist","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/utilities/deepspeed.py","lineNumber":41,"sourceCode":"from lightning.fabric.utilities.types import _PATH\nfrom lightning.pytorch.strategies.deepspeed import _DEEPSPEED_AVAILABLE\n\nCPU_DEVICE = torch.device(\"cpu\")\n\n\ndef ds_checkpoint_dir(checkpoint_dir: _PATH, tag: str | None = None) -> str:\n    if tag is None:\n        latest_path = os.path.join(checkpoint_dir, \"latest\")\n        if os.path.isfile(latest_path):\n            with open(latest_path) as fd:\n                tag = fd.read().strip()\n        else:\n            raise ValueError(f\"Unable to find 'latest' file at {latest_path}\")\n\n    directory = os.path.join(checkpoint_dir, tag)\n\n    if not os.path.isdir(directory):\n        raise FileNotFoundError(f\"Directory '{ds_checkpoint_dir}' doesn't exist\")\n    return directory\n\n\n# Modified script from https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/utils/zero_to_fp32.py\ndef convert_zero_checkpoint_to_fp32_state_dict(\n    checkpoint_dir: _PATH, output_file: _PATH, tag: str | None = None\n) -> dict[str, Any]:\n    \"\"\"Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be loaded with\n    ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed. It gets copied into the top\n    level checkpoint dir, so the user can easily do the conversion at any point in the future. Once extracted, the\n    weights don't require DeepSpeed and can be used in any application. Additionally the script has been modified to\n    ensure we keep the lightning state inside the state dict for being able to run\n    ``LightningModule.load_from_checkpoint('...')```.\n\n    Args:\n        checkpoint_dir: path to the desired checkpoint folder.\n            (one that contains the tag-folder, like ``global_step14``)\n        output_file: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/utilities/deepspeed.py#L23-L59","documentation":"Raised as FileNotFoundError by ds_checkpoint_dir when the resolved checkpoint tag directory (checkpoint_dir joined with the tag from the 'latest' file or the tag argument) does not exist on disk. The tag was obtained, but the folder holding the DeepSpeed sharded files is missing, so conversion cannot proceed.","triggerScenarios":"convert_zero_checkpoint_to_fp32_state_dict with a tag whose subdirectory isn't present, or a 'latest' file containing a stale tag; also relative-path issues where the working directory differs from the training run.","commonSituations":"'latest' points to a tag that was deleted or never synced; checkpoint directory moved between machines; typo in the tag argument; partial uploads of distributed checkpoints.","solutions":["Verify the tag directory exists: ls <checkpoint_dir>/<tag> and correct the tag argument","If 'latest' is stale, edit it to an existing tag or pass tag= explicitly","Confirm checkpoint_dir is an absolute path (avoid CWD-relative mismatches)"],"exampleFix":"# before\nconvert_zero_checkpoint_to_fp32_state_dict('ckpt', 'model.pt', tag='1000')  # ckpt/1000 missing\n\n# after\nimport os\ntags = [d for d in os.listdir('ckpt') if d.isdigit()]\nconvert_zero_checkpoint_to_fp32_state_dict('ckpt', 'model.pt', tag=tags[-1])","handlingStrategy":"validation","validationCode":"import os\nckpt_dir, tag = 'path/to/ckpt', '2000'\nif tag is None:\n    with open(os.path.join(ckpt_dir, 'latest')) as f:\n        tag = f.read().strip()\nassert os.path.isdir(os.path.join(ckpt_dir, tag)), f'tag dir {tag} missing in {ckpt_dir}'","typeGuard":null,"tryCatchPattern":"from lightning.pytorch.utilities.deepspeed import convert_zero_checkpoint_to_fp32_state_dict\ntry:\n    convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out, tag=tag)\nexcept FileNotFoundError:\n    tag = sorted(d for d in os.listdir(ckpt_dir) if d.isdigit())[-1]\n    convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, out, tag=tag)","preventionTips":["Use absolute paths for checkpoint directories to avoid CWD mismatches","Check the 'latest' file's tag matches an existing subdirectory after moving checkpoints","Validate the tag directory contains the expected sharded files before conversion"],"tags":["pytorch-lightning","deepspeed","checkpoint","file-not-found"],"backgroundTag":"deepspeed-checkpoint-directory-missing","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}