WZMIAOMIAO/deep-learning-for-image-processing · error · EnvironmentError

not support distributed training.

Error message

not support distributed training.

What it means

After calling init_distributed_mode(args), this DeepPose multi-GPU script checks args.distributed; if initialization did not actually enable distributed mode (init silently returns when env is not a distributed launch, e.g. no RANK/WORLD_SIZE env vars or --dist flag handling), it raises EnvironmentError because the rest of the script assumes DDP setup.

Source

Thrown at pytorch_keypoint/DeepPose/train_multi_GPU.py:44

    parser.add_argument("--batch_size", type=int, default=32, help="size of the batches")
    parser.add_argument("--num_workers", type=int, default=8, help="number of workers, default: 8")
    parser.add_argument("--num_keypoints", type=int, default=98, help="number of keypoints")
    parser.add_argument("--lr", type=float, default=5e-4, help="Adam: learning rate")
    parser.add_argument('--lr_steps', default=[170, 200], nargs='+', type=int,
                        help='decrease lr every step-size epochs')
    parser.add_argument("--warmup_epoch", type=int, default=10, help="number of warmup epoch for training")
    parser.add_argument('--resume', default='', type=str, help='resume from checkpoint')
    parser.add_argument('--dist-url', default='env://', help='url used to set up distributed training')
    parser.add_argument('--test_only', action="store_true", help='Only test the model')

    return parser


def main(args):
    torch.manual_seed(1234)
    init_distributed_mode(args)
    if not args.distributed:
        raise EnvironmentError("not support distributed training.")

    dataset_dir = args.dataset_dir
    save_weights_dir = args.save_weights_dir
    save_freq = args.save_freq
    eval_freq = args.eval_freq
    num_keypoints = args.num_keypoints
    num_workers = args.num_workers
    epochs = args.epochs
    bs = args.batch_size
    start_epoch = 0
    img_hw = args.img_hw
    device = torch.device(args.device)
    os.makedirs(save_weights_dir, exist_ok=True)

    # adjust learning rate
    args.lr = args.lr * args.world_size

    tb_writer = None

View on GitHub (pinned to 1ec3fe6f37)

Solutions

1. Launch with the distributed launcher: torchrun --nproc_per_node=2 train_multi_GPU.py (or torch.distributed.launch for older torch).
2. Confirm env vars RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT are set in the environment the script sees.
3. Check init_distributed_mode in train_utils/distributed_utils.py to see which flags/env it requires and pass them (e.g. --dist).
4. For single-GPU debugging, use a single-process train script instead of the DDP one.
5. If only one GPU, launch with --nproc_per_node=1 under the launcher so args.distributed is set.

Example fix

# before
python pytorch_keypoint/DeepPose/train_multi_GPU.py --num_keypoints 17
# EnvironmentError
# after
torchrun --nproc_per_node=2 pytorch_keypoint/DeepPose/train_multi_GPU.py --num_keypoints 17

import os
required = ["RANK", "WORLD_SIZE", "MASTER_ADDR", "MASTER_PORT"]
missing = [v for v in required if v not in os.environ]
if missing:
    raise SystemExit(f"Launch with torchrun; missing env: {missing}")

try:
    main(args)
except EnvironmentError as e:
    logging.error("%s — relaunch with: torchrun --nproc_per_node=N %s", e, sys.argv[0])
    sys.exit(2)

When it happens

Trigger: Running the script directly with python train_multi_GPU.py without a distributed launcher (torchrun / torch.distributed.launch), so init_distributed_mode never sets args.distributed=True; or passing flags that disable dist init; or init failing to read MASTER_ADDR/RANK env vars.

Common situations: Launching with plain python instead of torchrun --nproc_per_node=N; missing env vars inside a SLURM/K8s pod; old launch command syntax incompatible with installed torch version; intentionally running on one machine without launcher to debug.

Related errors

• not find GPU device for training.
• not find GPU device for training.
• not find GPU device for training.
• not find GPU device for training.
• not support data format '{self.data_format}'

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/a981cc7d01e2c6d6. Report an issue: GitHub.