unslothai/unsloth · error · ValueError

This run has {len(trainable)} trainable tensors and the chec

Error message

This run has {len(trainable)} trainable tensors and the checkpoint has {len(state)}: {'; '.join(detail)}. The LoRA configuration does not match the one it was saved from.

What it means

Beyond per-tensor shape checks, the resume path compares the SET of trainable tensor names in both directions. Counting only the checkpoint's tensors would miss a live parameter the checkpoint never had; a truncated or hand-edited adapter holding a strict subset previously passed, and the optimizer state then loaded Adam moments onto freshly initialized weights while reporting a clean resume. Any difference now fails with counts and up to three example names per direction.

Source

Thrown at studio/backend/core/training/diffusion_train_common.py:1870

                    f"Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run "
                    f"expects {tuple(p.shape)}; the LoRA configuration does not match."
                )
            p.copy_(saved.to(device = p.device, dtype = p.dtype))
            restored += 1
    # BOTH directions. Counting only the checkpoint's own tensors proves every saved tensor
    # landed somewhere; it says nothing about a live trainable parameter the checkpoint never
    # had. A truncated or hand-edited adapter file holding a strict SUBSET therefore passed,
    # and the full optimizer state was then loaded on top: restored Adam moments driving
    # freshly initialised weights, while the run reported a clean resume.
    unsaved = sorted(trainable - set(state))
    unknown = sorted(set(state) - trainable)
    if unsaved or unknown:
        detail = []
        if unsaved:
            detail.append(f"{len(unsaved)} not in the checkpoint (e.g. {', '.join(unsaved[:3])})")
        if unknown:
            detail.append(f"{len(unknown)} not in this run (e.g. {', '.join(unknown[:3])})")
        raise ValueError(
            f"This run has {len(trainable)} trainable tensors and the checkpoint has "
            f"{len(state)}: {'; '.join(detail)}. The LoRA configuration does not match the one "
            "it was saved from."
        )
    return restored


def _json_safe_progress(progress: Optional[dict[str, Any]]) -> dict[str, Any]:
    """Drop non-finite floats from the manifest's progress block. A diverged run pushes
    ``running_loss`` to NaN/inf, which json.dumps writes as the JS-only NaN/Infinity tokens --
    invalid strict JSON that a stricter reader (or a future consumer of these files) rejects."""
    out: dict[str, Any] = {}
    for key, value in (progress or {}).items():
        if isinstance(value, float) and not math.isfinite(value):
            continue
        out[key] = value
    return out

View on GitHub (pinned to 203007d190)

Solutions

1. Restore the exact lora_target_modules and base model used when the checkpoint was written (the manifest records them).
2. Start a new run if you intentionally changed the LoRA target set.
3. Discard hand-edited/truncated adapter files — they cannot be safely resumed even if shapes happen to line up.

Example fix

# before: checkpoint trained targets ['to_q','to_k'], resuming with extras
cfg = DiffusionLoraConfig(lora_target_modules=['to_q','to_k','to_v'], resume_from_checkpoint=ckpt)
# after
cfg = DiffusionLoraConfig(lora_target_modules=['to_q','to_k'], resume_from_checkpoint=ckpt)

saved_names = set(state.keys())
live_names = {k for k, _ in model.named_parameters() if k in getattr(model, '_trainable_names', saved_names)}
# simplest: compare counts before restoring
if len(saved_names) != sum(1 for _, p in model.named_parameters() if p.requires_grad and _ in saved_names):
    raise ValueError('trainable tensor set differs from checkpoint')

try:
    restore_trainable(model, state)
except ValueError as e:
    if 'trainable tensors and the checkpoint has' in str(e):
        recover_config_from_manifest_and_retry()  # or start fresh

When it happens

Trigger: Resuming with different lora_target_modules (extra names not in the checkpoint), a different base model revision exposing different layer names, or an adapter file that was truncated or hand-edited so some tensors are missing; conversely a checkpoint holding tensors the live model does not.

Common situations: Adding or removing target modules (e.g. adding 'ff.net' to the target list) between sessions; resuming a qwen-image run against a base revision that renamed modules; manually slimming a .safetensors adapter.

Related errors

• Checkpoint tensor '{name}' has shape {tuple(saved.shape)} bu
• The training checkpoint at '{directory}' could not be read;
• gradient_accumulation_steps must be >= 1
• lora_rank must be >= 1
• lora_alpha must be >= 1 (a zero/negative alpha scales the ad

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/75be607653178bc2. Report an issue: GitHub.