unslothai/unsloth · error · ValueError

Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu

Error message

Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run expects {tuple(p.shape)}; the LoRA configuration does not match.

What it means

When restoring a resume checkpoint, each saved trainable tensor must match the live parameter's shape by name before being copied in. A shape mismatch means the LoRA geometry changed between the save and the resume (different rank, different target modules producing different matrix shapes), so continuing is impossible.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:1851

    parameter INDEX it would still load cleanly -- leaving restored Adam moments and a restored
    LR position driving freshly initialised LoRA weights, while the run reports a normal resume.
    Raise instead."""
    if not state:
        return 0
    import torch

    restored = 0
    trainable: set[str] = set()
    with torch.no_grad():
        for name, p in model.named_parameters():
            if not p.requires_grad:
                continue
            trainable.add(name)
            saved = state.get(name)
            if saved is None:
                continue
            if tuple(saved.shape) != tuple(p.shape):
                raise ValueError(
                    f"Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run "
                    f"expects {tuple(p.shape)}; the LoRA configuration does not match."
                )
            p.copy_(saved.to(device = p.device, dtype = p.dtype))
            restored += 1
    # BOTH directions. Counting only the checkpoint's own tensors proves every saved tensor
    # landed somewhere; it says nothing about a live trainable parameter the checkpoint never
    # had. A truncated or hand-edited adapter file holding a strict SUBSET therefore passed,
    # and the full optimizer state was then loaded on top: restored Adam moments driving
    # freshly initialised weights, while the run reported a clean resume.
    unsaved = sorted(trainable - set(state))
    unknown = sorted(set(state) - trainable)
    if unsaved or unknown:
        detail = []
        if unsaved:
            detail.append(f"{len(unsaved)} not in the checkpoint (e.g. {', '.join(unsaved[:3])})")
        if unknown:
            detail.append(f"{len(unknown)} not in this run (e.g. {', '.join(unknown[:3])})")

View on GitHub (pinned to 203007d190)

Solutions

1. Recreate the exact LoRA configuration (rank, alpha default, target modules, base model) from the run that wrote the checkpoint, then resume.
2. If you intentionally changed the LoRA config, start a fresh run instead of resuming.
3. Check the checkpoint manifest (the writer stores the config) to recover the original rank/targets.

Example fix

# before: checkpoint saved with rank=32, resuming with rank=16
cfg = DiffusionLoraConfig(lora_rank=16, resume_from_checkpoint='runs/ckpt-500')
# after
cfg = DiffusionLoraConfig(lora_rank=32, resume_from_checkpoint='runs/ckpt-500')

ckpt_shapes = {k: tuple(v.shape) for k, v in torch.load(adapter_path).items()}
live_shapes = {k: tuple(p.shape) for k, p in model.named_parameters() if p.requires_grad}
for name, shape in live_shapes.items():
    saved = ckpt_shapes.get(name)
    if saved is not None and saved != shape:
        raise ValueError(f'LoRA config changed for {name}: {saved} vs {shape}')

try:
    restored = restore_trainable(model, state)
except ValueError as e:
    if 'LoRA configuration does not match' in str(e):
        start_fresh_run()  # config drifted; resume is impossible

When it happens

Trigger: resume_from_checkpoint pointing at an adapter saved with lora_rank=32 while the new run uses lora_rank=16 (or different lora_target_modules / base model), so e.g. lora_A tensors have shape (32, in_features) vs (16, in_features).

Common situations: User tunes hyperparameters between sessions and expects resume to follow; two runs writing checkpoints into a shared directory with different LoRA configs; resuming with a different base model whose layer widths differ.

Related errors

• This run has {len(trainable)} trainable tensors and the chec
• The training checkpoint at '{directory}' could not be read;
• This checkpoint does not record which optimizer wrote its st
• This checkpoint's optimizer state was written by {saved_opti
• Resume checkpoint must belong to a stopped or errored run wi

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/acc8e7e37a1421c4. Report an issue: GitHub.