{"record":{"id":"fa3fcdfca34d8291","repo":"unslothai/unsloth","slug":"this-checkpoint-s-optimizer-state-was-written-by","errorCode":null,"errorMessage":"This checkpoint's optimizer state was written by {saved_optimizer}, but this machine builds {live_optimizer}. Install the same optimizer backend (or unset UNSLOTH_DIFFUSION_FP32_OPTIM) to continue this run.","messagePattern":"This checkpoint's optimizer state was written by (.+?), but this machine builds (.+?)\\. Install the same optimizer backend \\(or unset UNSLOTH_DIFFUSION_FP32_OPTIM\\) to continue this run\\.","errorType":"exception","errorClass":"ResumeError","httpStatus":null,"severity":"error","filePath":"studio/backend/core/training/diffusion_train_common.py","lineNumber":2039,"sourceCode":"        # available, UNSLOTH_DIFFUSION_FP32_OPTIM), not from the config, so a checkpoint can\n        # legitimately arrive with foreign moments: AdamW8bit stores \"state1\"/\"state2\", torch\n        # AdamW stores \"exp_avg\"/\"exp_avg_sq\". Shapes and counts match, so load_state_dict\n        # accepts them and the first step dies on a bare KeyError. Refuse with a real reason.\n        saved_optimizer = ckpt.optimizer_class\n        live_optimizer = optimizer_key(optimizer)\n        if not saved_optimizer:\n            # This writer records the class whenever it writes moments, so an optimizer file\n            # with no class beside it is a hand-edited or half-written bundle -- and letting it\n            # through is the same failure the check below exists for: foreign moments load\n            # cleanly (shapes and counts match) and die on the first step, in the child, after\n            # the route preflight has already evicted the resident models.\n            raise ResumeError(\n                \"This checkpoint does not record which optimizer wrote its state, so its \"\n                \"moments cannot be safely restored. Resume from an earlier checkpoint, or \"\n                \"start a new run.\"\n            )\n        if saved_optimizer != live_optimizer:\n            raise ResumeError(\n                f\"This checkpoint's optimizer state was written by {saved_optimizer}, but this \"\n                f\"machine builds {live_optimizer}. Install the same optimizer backend (or unset \"\n                f\"UNSLOTH_DIFFUSION_FP32_OPTIM) to continue this run.\"\n            )\n        # Optimizer state is keyed by parameter POSITION, so load_state_dict rebinds the saved\n        # moments onto whatever order this process built. The adapter tensors above were restored\n        # by NAME, so a PEFT/diffusers upgrade that changes traversal order while keeping the same\n        # names leaves the two disagreeing: many LoRA projections share a shape, so every moment\n        # loads cleanly onto the wrong tensor and the continued trajectory is silently corrupt.\n        saved_names = ckpt.optimizer_param_names\n        live_names = list(trainable_state_dict(model))\n        if saved_names is not None and saved_names != live_names:\n            raise ResumeError(\n                \"This checkpoint's optimizer state was written for a different parameter order \"\n                \"than this build produces, so its moments cannot be matched to this run's \"\n                \"tensors. Start a new run, or resume on the version that wrote it.\"\n            )\n        # load_state_dict replaces the param groups too, so the checkpoint's learning rate wins","sourceCodeStart":2021,"sourceCodeEnd":2057,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/training/diffusion_train_common.py#L2021-L2057","documentation":"Resume raises ResumeError when the checkpoint's recorded optimizer_class differs from optimizer_key(optimizer) on this machine. The backend is chosen by environment (UNSLOTH_DIFFUSION_FP32_OPTIM), so a checkpoint legitimately written by AdamW8bit cannot be safely continued by torch AdamW: shapes and counts match, but the moment keys differ and the first training step crashes with a KeyError inside the child process.","triggerScenarios":"Checkpoint written with UNSLOTH_DIFFUSION_FP32_OPTIM set (AdamW8bit) then resumed with the variable unset (or vice versa); moving a run to a machine where bitsandbytes is absent so the optimizer falls back to torch AdamW.","commonSituations":"Environment differences between the original training host and the resume host; a .env change between sessions; uninstalling bitsandbytes and expecting old checkpoints to continue.","solutions":["Recreate the original optimizer environment: set (or unset) UNSLOTH_DIFFUSION_FP32_OPTIM to match what wrote the checkpoint.","Install the same optimizer backend (e.g. bitsandbytes) on the resuming machine.","If the backend is genuinely gone, start a new run — the foreign moments cannot be salvaged."],"exampleFix":"# before: checkpoint written with AdamW8bit, resumed without it\nunset UNSLOTH_DIFFUSION_FP32_OPTIM\n# after\nexport UNSLOTH_DIFFUSION_FP32_OPTIM=1  # match the value used when the checkpoint was saved","handlingStrategy":"validation","validationCode":"import json, os\nm = json.loads((ckpt_dir / 'manifest.json').read_text())\nsaved = m.get('optimizer_class')\nlive = 'AdamW8bit' if os.environ.get('UNSLOTH_DIFFUSION_FP32_OPTIM') else 'AdamW'\nif saved and saved != live:\n    raise ValueError(f'optimizer backend mismatch: checkpoint={saved}, machine={live}')","typeGuard":null,"tryCatchPattern":"try:\n    resume(run, checkpoint)\nexcept ResumeError as e:\n    if 'written by' in str(e) and 'optimizer' in str(e):\n        os.environ['UNSLOTH_DIFFUSION_FP32_OPTIM'] = '1'  # or unset, to match\n        rebuild_and_resume(run, checkpoint)","preventionTips":["Pin and record the optimizer environment variable in the run manifest; restore it on resume.","Keep the same optimizer backend installed on all machines that share a run."],"tags":["training","checkpoint","resume","optimizer","environment"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}