unslothai/unsloth · error · ResumeError

This checkpoint does not record which optimizer wrote its st

Error message

This checkpoint does not record which optimizer wrote its state, so its moments cannot be safely restored. Resume from an earlier checkpoint, or start a new run.

What it means

Resume raises ResumeError when the checkpoint manifest lacks optimizer_class. The writer records the optimizer class whenever it writes moments, so an optimizer state file with no class beside it is a hand-edited or half-written bundle. Different optimizer backends store differently-named moments (AdamW8bit: state1/state2; torch AdamW: exp_avg/exp_avg_sq) with matching shapes and counts, so load_state_dict would accept foreign moments and the first step would die on a bare KeyError — after the route preflight already evicted the resident models.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:2033

    )
    ckpt = load_checkpoint(path)
    load_trainable_state_dict(model, ckpt.tensors("adapter"))
    optimizer_state = ckpt.torch_state("optimizer")
    if optimizer_state is not None:
        # The trainers pick their optimizer from the HOST (bitsandbytes present, a fused kernel
        # available, UNSLOTH_DIFFUSION_FP32_OPTIM), not from the config, so a checkpoint can
        # legitimately arrive with foreign moments: AdamW8bit stores "state1"/"state2", torch
        # AdamW stores "exp_avg"/"exp_avg_sq". Shapes and counts match, so load_state_dict
        # accepts them and the first step dies on a bare KeyError. Refuse with a real reason.
        saved_optimizer = ckpt.optimizer_class
        live_optimizer = optimizer_key(optimizer)
        if not saved_optimizer:
            # This writer records the class whenever it writes moments, so an optimizer file
            # with no class beside it is a hand-edited or half-written bundle -- and letting it
            # through is the same failure the check below exists for: foreign moments load
            # cleanly (shapes and counts match) and die on the first step, in the child, after
            # the route preflight has already evicted the resident models.
            raise ResumeError(
                "This checkpoint does not record which optimizer wrote its state, so its "
                "moments cannot be safely restored. Resume from an earlier checkpoint, or "
                "start a new run."
            )
        if saved_optimizer != live_optimizer:
            raise ResumeError(
                f"This checkpoint's optimizer state was written by {saved_optimizer}, but this "
                f"machine builds {live_optimizer}. Install the same optimizer backend (or unset "
                f"UNSLOTH_DIFFUSION_FP32_OPTIM) to continue this run."
            )
        # Optimizer state is keyed by parameter POSITION, so load_state_dict rebinds the saved
        # moments onto whatever order this process built. The adapter tensors above were restored
        # by NAME, so a PEFT/diffusers upgrade that changes traversal order while keeping the same
        # names leaves the two disagreeing: many LoRA projections share a shape, so every moment
        # loads cleanly onto the wrong tensor and the continued trajectory is silently corrupt.
        saved_names = ckpt.optimizer_param_names
        live_names = list(trainable_state_dict(model))
        if saved_names is not None and saved_names != live_names:

View on GitHub (pinned to 203007d190)

Solutions

1. Resume from an earlier intact checkpoint in the same run directory.
2. Or start a new run if no intact checkpoint exists.
3. Verify the checkpoint bundle is complete (manifest fields plus state files all present and unmodified) before resuming.

Example fix

// before: resume from a half-written bundle
resume_from_checkpoint='runs/foo/step-500-partial'
// after: resume from the last complete bundle
resume_from_checkpoint='runs/foo/step-400'

import json
m = json.loads((ckpt_dir / 'manifest.json').read_text())
if 'optimizer_class' not in m or not m.get('optimizer_class'):
    raise ValueError('checkpoint manifest lacks optimizer_class; not safely resumable')

try:
    resume(run, checkpoint)
except ResumeError as e:
    if 'does not record which optimizer' in str(e):
        checkpoint = latest_intact_checkpoint(run.dir)  # step back to prior step
        resume(run, checkpoint)

When it happens

Trigger: resume_from_checkpoint pointing at a bundle whose manifest JSON has optimizer state files but no optimizer_class field: interrupted checkpoint writes, files copied between bundles, manifests edited by hand, or checkpoints written by an older writer version.

Common situations: Process killed mid-checkpoint; user reassembled a checkpoint directory from parts; partial upload/sync of a run directory.

Related errors

• This checkpoint's optimizer state was written by {saved_opti
• The training checkpoint at '{directory}' could not be read;
• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• This run has {len(trainable)} trainable tensors and the chec
• Resume checkpoint must belong to a stopped or errored run wi

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/d6c1a55d528b9794. Report an issue: GitHub.