unslothai/unsloth · error · ResumeError

The training checkpoint at '{directory}' could not be read;

Error message

The training checkpoint at '{directory}' could not be read; it may have been deleted or damaged since the run started.

What it means

Raised by load_checkpoint() when read_checkpoint() returns None for a resume bundle that preflight_resume had previously accepted. It means the checkpoint directory's manifest (or the torch.load-able role files behind it) became unreadable between preflight and load. Typical causes are a concurrent delete of the checkpoint directory or a network/half-mounted volume that went away mid-run. It is a ResumeError, so it specifically signals a broken resume rather than a bad initial training request.

Source

Thrown at studio/backend/core/training/diffusion_checkpoint.py:2192

        Loaded with ``weights_only = True``: these files are written by Studio into its own
        outputs directory, but a resume path is client-supplied, so the loader must never be
        able to execute pickled code. Verified to round-trip bitsandbytes AdamW8bit state,
        whose quantized moments and maps are plain uint8/fp32 tensors."""
        import torch

        path = self._file(role)
        if path is None:
            return None
        return torch.load(str(path), map_location = "cpu", weights_only = True)


def load_checkpoint(path: str | os.PathLike[str]) -> LoadedCheckpoint:
    """Open a bundle that ``preflight_resume`` already accepted. Raises ResumeError if it
    became unreadable in between (a concurrent delete, a half-mounted volume)."""
    directory = Path(path).expanduser()
    manifest = read_checkpoint(directory)
    if manifest is None:
        raise ResumeError(
            f"The training checkpoint at '{directory}' could not be read; it may have been "
            "deleted or damaged since the run started."
        )
    return LoadedCheckpoint(path = directory, manifest = manifest)

View on GitHub (pinned to 203007d190)

Solutions

1. Verify the checkpoint directory still exists and contains its manifest before retrying the resume (ls the checkpoint-<N> path).
2. If the bundle was deleted or damaged, resume from an earlier intact checkpoint-<M> if one exists, otherwise restart training from step 0.
3. Stop concurrent cleanup processes (cron jobs, retention scripts, other Studio sessions) that prune checkpoints while runs are live.
4. For network volumes, remount the storage and confirm the mount is fully healthy, then retry the resume once.

Example fix

# before
ckpt = load_checkpoint(resume_dir)  # dir deleted by a cleanup job after preflight

# after
from pathlib import Path
if not (Path(resume_dir) / "manifest.json").is_file():
    resume_dir = latest_intact_checkpoint(run_dir)  # or restart from scratch
ckpt = load_checkpoint(resume_dir)

from pathlib import Path

def checkpoint_readable(resume_dir: str) -> bool:
    d = Path(resume_dir).expanduser()
    return d.is_dir() and (d / "manifest.json").is_file()

from core.training.diffusion_checkpoint import ResumeError
try:
    loaded = load_checkpoint(resume_dir)
except ResumeError as e:
    # fall back to the newest intact bundle, or restart from step 0
    resume_dir = latest_intact_checkpoint(run_dir) or None

When it happens

Trigger: Calling the resume path (cfg.resume_from_checkpoint set) with a checkpoint-<N> bundle whose directory was deleted, moved, or corrupted after preflight_resume() returned OK; a manifest file that no longer parses; role files (adapter/optimizer/etc.) that torch.load cannot read under weights_only=True on a volume that partially mounted.

Common situations: A cleanup job or another Studio session deleting old checkpoints while a run is being resumed; NAS/S3-fuse mounts that drop mid-session; a user manually trimming the checkpoints folder between stopping a run and resuming it; disk-full during an earlier save leaving a half-written bundle that preflight tolerated.

Related errors

• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• This run has {len(trainable)} trainable tensors and the chec
• This checkpoint does not record which optimizer wrote its st
• This checkpoint's optimizer state was written by {saved_opti
• Resume checkpoint must belong to a stopped or errored run wi

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/8ae0ec4b5aff9b1f. Report an issue: GitHub.