{"record":{"id":"acc8e7e37a1421c4","repo":"unslothai/unsloth","slug":"checkpoint-tensor-name-has-shape-tuple-saved","errorCode":null,"errorMessage":"Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run expects {tuple(p.shape)}; the LoRA configuration does not match.","messagePattern":"Checkpoint tensor '(.+?)' has shape (.+?) but this run expects (.+?); the LoRA configuration does not match\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/diffusion_train_common.py","lineNumber":1851,"sourceCode":"    parameter INDEX it would still load cleanly -- leaving restored Adam moments and a restored\n    LR position driving freshly initialised LoRA weights, while the run reports a normal resume.\n    Raise instead.\"\"\"\n    if not state:\n        return 0\n    import torch\n\n    restored = 0\n    trainable: set[str] = set()\n    with torch.no_grad():\n        for name, p in model.named_parameters():\n            if not p.requires_grad:\n                continue\n            trainable.add(name)\n            saved = state.get(name)\n            if saved is None:\n                continue\n            if tuple(saved.shape) != tuple(p.shape):\n                raise ValueError(\n                    f\"Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run \"\n                    f\"expects {tuple(p.shape)}; the LoRA configuration does not match.\"\n                )\n            p.copy_(saved.to(device = p.device, dtype = p.dtype))\n            restored += 1\n    # BOTH directions. Counting only the checkpoint's own tensors proves every saved tensor\n    # landed somewhere; it says nothing about a live trainable parameter the checkpoint never\n    # had. A truncated or hand-edited adapter file holding a strict SUBSET therefore passed,\n    # and the full optimizer state was then loaded on top: restored Adam moments driving\n    # freshly initialised weights, while the run reported a clean resume.\n    unsaved = sorted(trainable - set(state))\n    unknown = sorted(set(state) - trainable)\n    if unsaved or unknown:\n        detail = []\n        if unsaved:\n            detail.append(f\"{len(unsaved)} not in the checkpoint (e.g. {', '.join(unsaved[:3])})\")\n        if unknown:\n            detail.append(f\"{len(unknown)} not in this run (e.g. {', '.join(unknown[:3])})\")","sourceCodeStart":1833,"sourceCodeEnd":1869,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/diffusion_train_common.py#L1833-L1869","documentation":"When restoring a resume checkpoint, each saved trainable tensor must match the live parameter's shape by name before being copied in. A shape mismatch means the LoRA geometry changed between the save and the resume (different rank, different target modules producing different matrix shapes), so continuing is impossible.","triggerScenarios":"resume_from_checkpoint pointing at an adapter saved with lora_rank=32 while the new run uses lora_rank=16 (or different lora_target_modules / base model), so e.g. lora_A tensors have shape (32, in_features) vs (16, in_features).","commonSituations":"User tunes hyperparameters between sessions and expects resume to follow; two runs writing checkpoints into a shared directory with different LoRA configs; resuming with a different base model whose layer widths differ.","solutions":["Recreate the exact LoRA configuration (rank, alpha default, target modules, base model) from the run that wrote the checkpoint, then resume.","If you intentionally changed the LoRA config, start a fresh run instead of resuming.","Check the checkpoint manifest (the writer stores the config) to recover the original rank/targets."],"exampleFix":"# before: checkpoint saved with rank=32, resuming with rank=16\ncfg = DiffusionLoraConfig(lora_rank=16, resume_from_checkpoint='runs/ckpt-500')\n# after\ncfg = DiffusionLoraConfig(lora_rank=32, resume_from_checkpoint='runs/ckpt-500')","handlingStrategy":"validation","validationCode":"ckpt_shapes = {k: tuple(v.shape) for k, v in torch.load(adapter_path).items()}\nlive_shapes = {k: tuple(p.shape) for k, p in model.named_parameters() if p.requires_grad}\nfor name, shape in live_shapes.items():\n    saved = ckpt_shapes.get(name)\n    if saved is not None and saved != shape:\n        raise ValueError(f'LoRA config changed for {name}: {saved} vs {shape}')","typeGuard":null,"tryCatchPattern":"try:\n    restored = restore_trainable(model, state)\nexcept ValueError as e:\n    if 'LoRA configuration does not match' in str(e):\n        start_fresh_run()  # config drifted; resume is impossible","preventionTips":["Persist the full LoRA config with each checkpoint and reload it verbatim on resume.","Treat any hyperparameter change as a new run; never resume across config changes."],"tags":["training","checkpoint","resume","lora","shape-mismatch"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}