hiyouga/LlamaFactory · critical · RuntimeError

Unexpected missing keys when loading checkpoint model weight

Error message

Unexpected missing keys when loading checkpoint model weights: {incompatible_keys.missing_keys}.

What it means

During checkpoint resume, non-adapter weights are loaded with strict=False, but any missing keys in incompatible_keys.missing_keys are treated as fatal: the checkpoint state_dict does not cover the full model, so silently continuing would train with randomly initialized modules. Typical causes are resuming a checkpoint saved from a different architecture or a partially saved/rotated checkpoint.

Source

Thrown at src/llamafactory/v1/core/utils/checkpoint.py:217

        adapter_file = os.path.join(model_dir, "adapter_model.safetensors")
        if not os.path.exists(adapter_file):
            adapter_file = os.path.join(model_dir, "adapter_model.bin")
            adapter_state = torch.load(adapter_file, map_location="cpu", weights_only=True)
        else:
            adapter_state = load_file(adapter_file, device="cpu")
        set_peft_model_state_dict(model_to_load, adapter_state)
    else:
        state_dict = {}
        for f in sorted(glob.glob(os.path.join(model_dir, "*.safetensors"))):
            state_dict.update(load_file(f, device="cpu"))
        if not state_dict:
            for f in sorted(glob.glob(os.path.join(model_dir, "*.bin"))):
                state_dict.update(torch.load(f, map_location="cpu", weights_only=True))
        if state_dict:
            incompatible_keys = model_to_load.load_state_dict(state_dict, strict=False)
            if incompatible_keys.missing_keys:
                raise RuntimeError(
                    f"Unexpected missing keys when loading checkpoint model weights: {incompatible_keys.missing_keys}."
                )
        else:
            logger.warning_rank0(f"No model weights found in {model_dir}, skipping model state restore.")

    optim_path = os.path.join(ckpt_dir, "optimizer", "state_dict.pt")
    if os.path.exists(optim_path):
        optimizer.load_state_dict(torch.load(optim_path, map_location=map_location, weights_only=True))


class TrainingCheckpointCoordinator:
    """Coordinates full checkpoint save/resume for a trainer instance."""

    def __init__(self, trainer: Any) -> None:
        self._t = trainer

    @property
    def _dist_name(self) -> str | None:

View on GitHub (pinned to f28afaf635)

Solutions

1. Resume with the exact same base model/config used when the checkpoint was created
2. Inspect the missing keys: if they are all in one subsystem (e.g. vision tower), the checkpoint/model pairing is wrong — fix the pairing rather than the code
3. Verify all shards are present in the checkpoint directory (compare against the save-time file list)
4. If starting fresh, clear or move the stale output_dir instead of resuming

Example fix

# before
trainer resume with --model_name_or_path Qwen/Qwen3-8B  # ckpt saved from Qwen3-4B -> missing keys

# after
trainer resume with --model_name_or_path Qwen/Qwen3-4B  # identical to the checkpointed run

def checkpoint_covers_model(model_dir: str, model) -> bool:
    sd = {}
    import glob, safetensors.torch as st
    for f in glob.glob(os.path.join(model_dir, "*.safetensors")):
        sd.update(st.load_file(f))
    return set(model.state_dict().keys()) <= set(sd.keys())

try:
    load_checkpoint_state(model, None, ckpt_dir)
except RuntimeError as e:
    if "missing keys" in str(e):
        log_and_abort_resume(ckpt_dir)  # never train on with random modules

When it happens

Trigger: load_checkpoint_state with a model_dir whose safetensors/.bin files were saved from a different base model (layer counts, vocab size, tied embeddings) or a checkpoint whose weight files are incomplete (rotation deleted a shard, interrupted save).

Common situations: Changing model_name_or_path between runs but reusing output_dir; resuming after a crashed save; checkpoints saved with shard subsets; vision-tower keys missing when the processor config changed.

Related errors

• Output directory already exists and is not empty. Please set
• `resume_from_checkpoint` will be supported in the future ver
• Checkpoint directory does not exist: {ckpt_dir}
• FLA operator `{op_name}` did not match any model module attr
• No checkpoint files found in {hf_model_path}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/1b65fe704f742a38. Report an issue: GitHub.