WZMIAOMIAO/deep-learning-for-image-processing · error · KeyError

%s is not compatible with %s. Specify --weights '' or specif

Error message

%s is not compatible with %s. Specify --weights '' or specify a --cfg compatible with %s. See https://github.com/ultralytics/yolov3/issues/657

What it means

Same guard as train.py: when loading a checkpoint in multi-GPU main(), mismatched checkpoint keys cause model.state_dict()[k] to raise KeyError, re-raised as a KeyError message saying the weights and cfg are incompatible, with a link to yolov3 issue #657.

Source

Thrown at pytorch_object_detection/yolov3_spp/train_multi_GPU.py:96

    # Initialize model
    model = Darknet(cfg).to(device)

    start_epoch = 0
    best_map = 0.0
    # 如果指定了预训练权重，则载入预训练权重
    if weights.endswith(".pt"):
        ckpt = torch.load(weights, map_location=device)

        # load model
        try:
            ckpt["model"] = {k: v for k, v in ckpt["model"].items()
                             if model.state_dict()[k].numel() == v.numel()}
            model.load_state_dict(ckpt["model"], strict=False)
        except KeyError as e:
            s = "%s is not compatible with %s. Specify --weights '' or specify a --cfg compatible with %s. " \
                "See https://github.com/ultralytics/yolov3/issues/657" % (opt.weights, opt.cfg, opt.weights)
            raise KeyError(s) from e

        if opt.rank in [-1, 0]:
            # load results
            if ckpt.get("training_results") is not None:
                with open(results_file, "w") as file:
                    file.write(ckpt["training_results"])  # write results.txt

        # epochs
        start_epoch = ckpt["epoch"] + 1
        if epochs < start_epoch:
            print('%s has been trained for %g epochs. Fine-tuning for %g additional epochs.' %
                  (opt.weights, ckpt['epoch'], epochs))
            epochs += ckpt['epoch']  # finetune additional epochs

        if opt.amp and "scaler" in ckpt:
            scaler.load_state_dict(ckpt["scaler"])

        del ckpt

View on GitHub (pinned to 1ec3fe6f37)

Solutions

Pass --weights '' to train from scratch with the current cfg
Use a --cfg that matches the checkpoint architecture exactly
Preprocess the checkpoint to drop/resize incompatible layers before loading

Example fix

// before
python train_multi_GPU.py --cfg cfg/yolov3-tiny3.cfg --weights weights/yolov3-spp.pt
// after
python train_multi_GPU.py --cfg cfg/yolov3-tiny3.cfg --weights ''
# or match the checkpoint: --cfg cfg/yolov3-spp.cfg --weights weights/yolov3-spp.pt

Defensive patterns

Strategy: try-catch

Validate before calling

import torch
ckpt = torch.load(opt.weights, map_location='cpu')
missing = set(ckpt['model'].keys()) - set(model.state_dict().keys())
assert not missing, f'weights/cfg mismatch, e.g. {sorted(missing)[:3]}'

Try / catch

try:
    model.load_state_dict(ckpt['model'], strict=False)
except KeyError as e:
    raise SystemExit('weights/cfg mismatch in multi-GPU run: use --weights \'\'' or the matching cfg') from e

Prevention

Store cfg + hyp + weights together per run and resume only from matching combos
Re-generate checkpoints after any cfg architecture change
Pre-validate checkpoint keys against the model state_dict before DDP launch

When it happens

Trigger: Running train_multi_GPU.py where --weights checkpoint architecture does not match --cfg (class count, depth/width multiples, or SPP layers differ).

Common situations: Distributed fine-tune of a COCO checkpoint with a custom-N-classes cfg; resuming a run whose cfg was edited between runs; using single-GPU weights with a modified multi-GPU cfg.

Related errors

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/1dca657eceb96702. Report an issue: GitHub.