unslothai/unsloth · error · ResumeError

This checkpoint's optimizer state was written by {saved_opti

Error message

This checkpoint's optimizer state was written by {saved_optimizer}, but this machine builds {live_optimizer}. Install the same optimizer backend (or unset UNSLOTH_DIFFUSION_FP32_OPTIM) to continue this run.

What it means

Resume raises ResumeError when the checkpoint's recorded optimizer_class differs from optimizer_key(optimizer) on this machine. The backend is chosen by environment (UNSLOTH_DIFFUSION_FP32_OPTIM), so a checkpoint legitimately written by AdamW8bit cannot be safely continued by torch AdamW: shapes and counts match, but the moment keys differ and the first training step crashes with a KeyError inside the child process.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:2039

        # available, UNSLOTH_DIFFUSION_FP32_OPTIM), not from the config, so a checkpoint can
        # legitimately arrive with foreign moments: AdamW8bit stores "state1"/"state2", torch
        # AdamW stores "exp_avg"/"exp_avg_sq". Shapes and counts match, so load_state_dict
        # accepts them and the first step dies on a bare KeyError. Refuse with a real reason.
        saved_optimizer = ckpt.optimizer_class
        live_optimizer = optimizer_key(optimizer)
        if not saved_optimizer:
            # This writer records the class whenever it writes moments, so an optimizer file
            # with no class beside it is a hand-edited or half-written bundle -- and letting it
            # through is the same failure the check below exists for: foreign moments load
            # cleanly (shapes and counts match) and die on the first step, in the child, after
            # the route preflight has already evicted the resident models.
            raise ResumeError(
                "This checkpoint does not record which optimizer wrote its state, so its "
                "moments cannot be safely restored. Resume from an earlier checkpoint, or "
                "start a new run."
            )
        if saved_optimizer != live_optimizer:
            raise ResumeError(
                f"This checkpoint's optimizer state was written by {saved_optimizer}, but this "
                f"machine builds {live_optimizer}. Install the same optimizer backend (or unset "
                f"UNSLOTH_DIFFUSION_FP32_OPTIM) to continue this run."
            )
        # Optimizer state is keyed by parameter POSITION, so load_state_dict rebinds the saved
        # moments onto whatever order this process built. The adapter tensors above were restored
        # by NAME, so a PEFT/diffusers upgrade that changes traversal order while keeping the same
        # names leaves the two disagreeing: many LoRA projections share a shape, so every moment
        # loads cleanly onto the wrong tensor and the continued trajectory is silently corrupt.
        saved_names = ckpt.optimizer_param_names
        live_names = list(trainable_state_dict(model))
        if saved_names is not None and saved_names != live_names:
            raise ResumeError(
                "This checkpoint's optimizer state was written for a different parameter order "
                "than this build produces, so its moments cannot be matched to this run's "
                "tensors. Start a new run, or resume on the version that wrote it."
            )
        # load_state_dict replaces the param groups too, so the checkpoint's learning rate wins

View on GitHub (pinned to 203007d190)

Solutions

1. Recreate the original optimizer environment: set (or unset) UNSLOTH_DIFFUSION_FP32_OPTIM to match what wrote the checkpoint.
2. Install the same optimizer backend (e.g. bitsandbytes) on the resuming machine.
3. If the backend is genuinely gone, start a new run — the foreign moments cannot be salvaged.

Example fix

# before: checkpoint written with AdamW8bit, resumed without it
unset UNSLOTH_DIFFUSION_FP32_OPTIM
# after
export UNSLOTH_DIFFUSION_FP32_OPTIM=1  # match the value used when the checkpoint was saved

import json, os
m = json.loads((ckpt_dir / 'manifest.json').read_text())
saved = m.get('optimizer_class')
live = 'AdamW8bit' if os.environ.get('UNSLOTH_DIFFUSION_FP32_OPTIM') else 'AdamW'
if saved and saved != live:
    raise ValueError(f'optimizer backend mismatch: checkpoint={saved}, machine={live}')

try:
    resume(run, checkpoint)
except ResumeError as e:
    if 'written by' in str(e) and 'optimizer' in str(e):
        os.environ['UNSLOTH_DIFFUSION_FP32_OPTIM'] = '1'  # or unset, to match
        rebuild_and_resume(run, checkpoint)

When it happens

Trigger: Checkpoint written with UNSLOTH_DIFFUSION_FP32_OPTIM set (AdamW8bit) then resumed with the variable unset (or vice versa); moving a run to a machine where bitsandbytes is absent so the optimizer falls back to torch AdamW.

Common situations: Environment differences between the original training host and the resume host; a .env change between sessions; uninstalling bitsandbytes and expecting old checkpoints to continue.

Related errors

• This checkpoint does not record which optimizer wrote its st
• The training checkpoint at '{directory}' could not be read;
• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• This run has {len(trainable)} trainable tensors and the chec
• Resume checkpoint must belong to a stopped or errored run wi

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/fa3fcdfca34d8291. Report an issue: GitHub.