{"record":{"id":"75be607653178bc2","repo":"unslothai/unsloth","slug":"this-run-has-len-trainable-trainable-tensors-an","errorCode":null,"errorMessage":"This run has {len(trainable)} trainable tensors and the checkpoint has {len(state)}: {'; '.join(detail)}. The LoRA configuration does not match the one it was saved from.","messagePattern":"This run has (.+?) trainable tensors and the checkpoint has (.+?): (.+?)\\. The LoRA configuration does not match the one it was saved from\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/diffusion_train_common.py","lineNumber":1870,"sourceCode":"                    f\"Checkpoint tensor '{name}' has shape {tuple(saved.shape)} but this run \"\n                    f\"expects {tuple(p.shape)}; the LoRA configuration does not match.\"\n                )\n            p.copy_(saved.to(device = p.device, dtype = p.dtype))\n            restored += 1\n    # BOTH directions. Counting only the checkpoint's own tensors proves every saved tensor\n    # landed somewhere; it says nothing about a live trainable parameter the checkpoint never\n    # had. A truncated or hand-edited adapter file holding a strict SUBSET therefore passed,\n    # and the full optimizer state was then loaded on top: restored Adam moments driving\n    # freshly initialised weights, while the run reported a clean resume.\n    unsaved = sorted(trainable - set(state))\n    unknown = sorted(set(state) - trainable)\n    if unsaved or unknown:\n        detail = []\n        if unsaved:\n            detail.append(f\"{len(unsaved)} not in the checkpoint (e.g. {', '.join(unsaved[:3])})\")\n        if unknown:\n            detail.append(f\"{len(unknown)} not in this run (e.g. {', '.join(unknown[:3])})\")\n        raise ValueError(\n            f\"This run has {len(trainable)} trainable tensors and the checkpoint has \"\n            f\"{len(state)}: {'; '.join(detail)}. The LoRA configuration does not match the one \"\n            \"it was saved from.\"\n        )\n    return restored\n\n\ndef _json_safe_progress(progress: Optional[dict[str, Any]]) -> dict[str, Any]:\n    \"\"\"Drop non-finite floats from the manifest's progress block. A diverged run pushes\n    ``running_loss`` to NaN/inf, which json.dumps writes as the JS-only NaN/Infinity tokens --\n    invalid strict JSON that a stricter reader (or a future consumer of these files) rejects.\"\"\"\n    out: dict[str, Any] = {}\n    for key, value in (progress or {}).items():\n        if isinstance(value, float) and not math.isfinite(value):\n            continue\n        out[key] = value\n    return out\n","sourceCodeStart":1852,"sourceCodeEnd":1888,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/diffusion_train_common.py#L1852-L1888","documentation":"Beyond per-tensor shape checks, the resume path compares the SET of trainable tensor names in both directions. Counting only the checkpoint's tensors would miss a live parameter the checkpoint never had; a truncated or hand-edited adapter holding a strict subset previously passed, and the optimizer state then loaded Adam moments onto freshly initialized weights while reporting a clean resume. Any difference now fails with counts and up to three example names per direction.","triggerScenarios":"Resuming with different lora_target_modules (extra names not in the checkpoint), a different base model revision exposing different layer names, or an adapter file that was truncated or hand-edited so some tensors are missing; conversely a checkpoint holding tensors the live model does not.","commonSituations":"Adding or removing target modules (e.g. adding 'ff.net' to the target list) between sessions; resuming a qwen-image run against a base revision that renamed modules; manually slimming a .safetensors adapter.","solutions":["Restore the exact lora_target_modules and base model used when the checkpoint was written (the manifest records them).","Start a new run if you intentionally changed the LoRA target set.","Discard hand-edited/truncated adapter files — they cannot be safely resumed even if shapes happen to line up."],"exampleFix":"# before: checkpoint trained targets ['to_q','to_k'], resuming with extras\ncfg = DiffusionLoraConfig(lora_target_modules=['to_q','to_k','to_v'], resume_from_checkpoint=ckpt)\n# after\ncfg = DiffusionLoraConfig(lora_target_modules=['to_q','to_k'], resume_from_checkpoint=ckpt)","handlingStrategy":"validation","validationCode":"saved_names = set(state.keys())\nlive_names = {k for k, _ in model.named_parameters() if k in getattr(model, '_trainable_names', saved_names)}\n# simplest: compare counts before restoring\nif len(saved_names) != sum(1 for _, p in model.named_parameters() if p.requires_grad and _ in saved_names):\n    raise ValueError('trainable tensor set differs from checkpoint')","typeGuard":null,"tryCatchPattern":"try:\n    restore_trainable(model, state)\nexcept ValueError as e:\n    if 'trainable tensors and the checkpoint has' in str(e):\n        recover_config_from_manifest_and_retry()  # or start fresh","preventionTips":["Freeze lora_target_modules and base model identity per run; store them in the run manifest.","Never hand-edit adapter safetensors; regenerate from a clean save."],"tags":["training","checkpoint","resume","lora","validation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}