{"record":{"id":"8ae0ec4b5aff9b1f","repo":"unslothai/unsloth","slug":"the-training-checkpoint-at-directory-could-not","errorCode":null,"errorMessage":"The training checkpoint at '{directory}' could not be read; it may have been deleted or damaged since the run started.","messagePattern":"The training checkpoint at '(.+?)' could not be read; it may have been deleted or damaged since the run started\\.","errorType":"exception","errorClass":"ResumeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/diffusion_checkpoint.py","lineNumber":2192,"sourceCode":"        Loaded with ``weights_only = True``: these files are written by Studio into its own\n        outputs directory, but a resume path is client-supplied, so the loader must never be\n        able to execute pickled code. Verified to round-trip bitsandbytes AdamW8bit state,\n        whose quantized moments and maps are plain uint8/fp32 tensors.\"\"\"\n        import torch\n\n        path = self._file(role)\n        if path is None:\n            return None\n        return torch.load(str(path), map_location = \"cpu\", weights_only = True)\n\n\ndef load_checkpoint(path: str | os.PathLike[str]) -> LoadedCheckpoint:\n    \"\"\"Open a bundle that ``preflight_resume`` already accepted. Raises ResumeError if it\n    became unreadable in between (a concurrent delete, a half-mounted volume).\"\"\"\n    directory = Path(path).expanduser()\n    manifest = read_checkpoint(directory)\n    if manifest is None:\n        raise ResumeError(\n            f\"The training checkpoint at '{directory}' could not be read; it may have been \"\n            \"deleted or damaged since the run started.\"\n        )\n    return LoadedCheckpoint(path = directory, manifest = manifest)\n","sourceCodeStart":2174,"sourceCodeEnd":2197,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/diffusion_checkpoint.py#L2174-L2197","documentation":"Raised by load_checkpoint() when read_checkpoint() returns None for a resume bundle that preflight_resume had previously accepted. It means the checkpoint directory's manifest (or the torch.load-able role files behind it) became unreadable between preflight and load. Typical causes are a concurrent delete of the checkpoint directory or a network/half-mounted volume that went away mid-run. It is a ResumeError, so it specifically signals a broken resume rather than a bad initial training request.","triggerScenarios":"Calling the resume path (cfg.resume_from_checkpoint set) with a checkpoint-<N> bundle whose directory was deleted, moved, or corrupted after preflight_resume() returned OK; a manifest file that no longer parses; role files (adapter/optimizer/etc.) that torch.load cannot read under weights_only=True on a volume that partially mounted.","commonSituations":"A cleanup job or another Studio session deleting old checkpoints while a run is being resumed; NAS/S3-fuse mounts that drop mid-session; a user manually trimming the checkpoints folder between stopping a run and resuming it; disk-full during an earlier save leaving a half-written bundle that preflight tolerated.","solutions":["Verify the checkpoint directory still exists and contains its manifest before retrying the resume (ls the checkpoint-<N> path).","If the bundle was deleted or damaged, resume from an earlier intact checkpoint-<M> if one exists, otherwise restart training from step 0.","Stop concurrent cleanup processes (cron jobs, retention scripts, other Studio sessions) that prune checkpoints while runs are live.","For network volumes, remount the storage and confirm the mount is fully healthy, then retry the resume once."],"exampleFix":"# before\nckpt = load_checkpoint(resume_dir)  # dir deleted by a cleanup job after preflight\n\n# after\nfrom pathlib import Path\nif not (Path(resume_dir) / \"manifest.json\").is_file():\n    resume_dir = latest_intact_checkpoint(run_dir)  # or restart from scratch\nckpt = load_checkpoint(resume_dir)","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef checkpoint_readable(resume_dir: str) -> bool:\n    d = Path(resume_dir).expanduser()\n    return d.is_dir() and (d / \"manifest.json\").is_file()","typeGuard":null,"tryCatchPattern":"from core.training.diffusion_checkpoint import ResumeError\ntry:\n    loaded = load_checkpoint(resume_dir)\nexcept ResumeError as e:\n    # fall back to the newest intact bundle, or restart from step 0\n    resume_dir = latest_intact_checkpoint(run_dir) or None","preventionTips":["Never prune checkpoint directories while a run is live; schedule retention for idle periods.","Keep at least two checkpoint bundles per run so one damaged bundle is survivable.","On network volumes, verify the mount is healthy before resuming."],"tags":["checkpoint","resume","filesystem","training"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}