{"record":{"id":"1b65fe704f742a38","repo":"hiyouga/LlamaFactory","slug":"unexpected-missing-keys-when-loading-checkpoint-mo","errorCode":null,"errorMessage":"Unexpected missing keys when loading checkpoint model weights: {incompatible_keys.missing_keys}.","messagePattern":"Unexpected missing keys when loading checkpoint model weights: (.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/v1/core/utils/checkpoint.py","lineNumber":217,"sourceCode":"\n        adapter_file = os.path.join(model_dir, \"adapter_model.safetensors\")\n        if not os.path.exists(adapter_file):\n            adapter_file = os.path.join(model_dir, \"adapter_model.bin\")\n            adapter_state = torch.load(adapter_file, map_location=\"cpu\", weights_only=True)\n        else:\n            adapter_state = load_file(adapter_file, device=\"cpu\")\n        set_peft_model_state_dict(model_to_load, adapter_state)\n    else:\n        state_dict = {}\n        for f in sorted(glob.glob(os.path.join(model_dir, \"*.safetensors\"))):\n            state_dict.update(load_file(f, device=\"cpu\"))\n        if not state_dict:\n            for f in sorted(glob.glob(os.path.join(model_dir, \"*.bin\"))):\n                state_dict.update(torch.load(f, map_location=\"cpu\", weights_only=True))\n        if state_dict:\n            incompatible_keys = model_to_load.load_state_dict(state_dict, strict=False)\n            if incompatible_keys.missing_keys:\n                raise RuntimeError(\n                    f\"Unexpected missing keys when loading checkpoint model weights: {incompatible_keys.missing_keys}.\"\n                )\n        else:\n            logger.warning_rank0(f\"No model weights found in {model_dir}, skipping model state restore.\")\n\n    optim_path = os.path.join(ckpt_dir, \"optimizer\", \"state_dict.pt\")\n    if os.path.exists(optim_path):\n        optimizer.load_state_dict(torch.load(optim_path, map_location=map_location, weights_only=True))\n\n\nclass TrainingCheckpointCoordinator:\n    \"\"\"Coordinates full checkpoint save/resume for a trainer instance.\"\"\"\n\n    def __init__(self, trainer: Any) -> None:\n        self._t = trainer\n\n    @property\n    def _dist_name(self) -> str | None:","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/utils/checkpoint.py#L199-L235","documentation":"During checkpoint resume, non-adapter weights are loaded with strict=False, but any missing keys in incompatible_keys.missing_keys are treated as fatal: the checkpoint state_dict does not cover the full model, so silently continuing would train with randomly initialized modules. Typical causes are resuming a checkpoint saved from a different architecture or a partially saved/rotated checkpoint.","triggerScenarios":"load_checkpoint_state with a model_dir whose safetensors/.bin files were saved from a different base model (layer counts, vocab size, tied embeddings) or a checkpoint whose weight files are incomplete (rotation deleted a shard, interrupted save).","commonSituations":"Changing model_name_or_path between runs but reusing output_dir; resuming after a crashed save; checkpoints saved with shard subsets; vision-tower keys missing when the processor config changed.","solutions":["Resume with the exact same base model/config used when the checkpoint was created","Inspect the missing keys: if they are all in one subsystem (e.g. vision tower), the checkpoint/model pairing is wrong — fix the pairing rather than the code","Verify all shards are present in the checkpoint directory (compare against the save-time file list)","If starting fresh, clear or move the stale output_dir instead of resuming"],"exampleFix":"# before\ntrainer resume with --model_name_or_path Qwen/Qwen3-8B  # ckpt saved from Qwen3-4B -> missing keys\n\n# after\ntrainer resume with --model_name_or_path Qwen/Qwen3-4B  # identical to the checkpointed run","handlingStrategy":"try-catch","validationCode":"def checkpoint_covers_model(model_dir: str, model) -> bool:\n    sd = {}\n    import glob, safetensors.torch as st\n    for f in glob.glob(os.path.join(model_dir, \"*.safetensors\")):\n        sd.update(st.load_file(f))\n    return set(model.state_dict().keys()) <= set(sd.keys())","typeGuard":null,"tryCatchPattern":"try:\n    load_checkpoint_state(model, None, ckpt_dir)\nexcept RuntimeError as e:\n    if \"missing keys\" in str(e):\n        log_and_abort_resume(ckpt_dir)  # never train on with random modules","preventionTips":["Pin model_name_or_path in resumed runs to the checkpointed run's base model","Record base model identity in checkpoint metadata at save time and compare at resume"],"tags":["checkpoint","resume","state-dict","model-mismatch"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}