{"record":{"id":"a981cc7d01e2c6d6","repo":"WZMIAOMIAO/deep-learning-for-image-processing","slug":"not-support-distributed-training","errorCode":null,"errorMessage":"not support distributed training.","messagePattern":"not support distributed training\\.","errorType":"exception","errorClass":"EnvironmentError","httpStatus":null,"severity":"error","filePath":"pytorch_keypoint/DeepPose/train_multi_GPU.py","lineNumber":44,"sourceCode":"    parser.add_argument(\"--batch_size\", type=int, default=32, help=\"size of the batches\")\n    parser.add_argument(\"--num_workers\", type=int, default=8, help=\"number of workers, default: 8\")\n    parser.add_argument(\"--num_keypoints\", type=int, default=98, help=\"number of keypoints\")\n    parser.add_argument(\"--lr\", type=float, default=5e-4, help=\"Adam: learning rate\")\n    parser.add_argument('--lr_steps', default=[170, 200], nargs='+', type=int,\n                        help='decrease lr every step-size epochs')\n    parser.add_argument(\"--warmup_epoch\", type=int, default=10, help=\"number of warmup epoch for training\")\n    parser.add_argument('--resume', default='', type=str, help='resume from checkpoint')\n    parser.add_argument('--dist-url', default='env://', help='url used to set up distributed training')\n    parser.add_argument('--test_only', action=\"store_true\", help='Only test the model')\n\n    return parser\n\n\ndef main(args):\n    torch.manual_seed(1234)\n    init_distributed_mode(args)\n    if not args.distributed:\n        raise EnvironmentError(\"not support distributed training.\")\n\n    dataset_dir = args.dataset_dir\n    save_weights_dir = args.save_weights_dir\n    save_freq = args.save_freq\n    eval_freq = args.eval_freq\n    num_keypoints = args.num_keypoints\n    num_workers = args.num_workers\n    epochs = args.epochs\n    bs = args.batch_size\n    start_epoch = 0\n    img_hw = args.img_hw\n    device = torch.device(args.device)\n    os.makedirs(save_weights_dir, exist_ok=True)\n\n    # adjust learning rate\n    args.lr = args.lr * args.world_size\n\n    tb_writer = None","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/WZMIAOMIAO/deep-learning-for-image-processing/blob/1ec3fe6f374fc9969973a61f819de25658595afa/pytorch_keypoint/DeepPose/train_multi_GPU.py#L26-L62","documentation":"After calling init_distributed_mode(args), this DeepPose multi-GPU script checks args.distributed; if initialization did not actually enable distributed mode (init silently returns when env is not a distributed launch, e.g. no RANK/WORLD_SIZE env vars or --dist flag handling), it raises EnvironmentError because the rest of the script assumes DDP setup.","triggerScenarios":"Running the script directly with python train_multi_GPU.py without a distributed launcher (torchrun / torch.distributed.launch), so init_distributed_mode never sets args.distributed=True; or passing flags that disable dist init; or init failing to read MASTER_ADDR/RANK env vars.","commonSituations":"Launching with plain python instead of torchrun --nproc_per_node=N; missing env vars inside a SLURM/K8s pod; old launch command syntax incompatible with installed torch version; intentionally running on one machine without launcher to debug.","solutions":["Launch with the distributed launcher: torchrun --nproc_per_node=2 train_multi_GPU.py (or torch.distributed.launch for older torch).","Confirm env vars RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT are set in the environment the script sees.","Check init_distributed_mode in train_utils/distributed_utils.py to see which flags/env it requires and pass them (e.g. --dist).","For single-GPU debugging, use a single-process train script instead of the DDP one.","If only one GPU, launch with --nproc_per_node=1 under the launcher so args.distributed is set."],"exampleFix":"# before\npython pytorch_keypoint/DeepPose/train_multi_GPU.py --num_keypoints 17\n# EnvironmentError\n# after\ntorchrun --nproc_per_node=2 pytorch_keypoint/DeepPose/train_multi_GPU.py --num_keypoints 17","handlingStrategy":"validation","validationCode":"import os\nrequired = [\"RANK\", \"WORLD_SIZE\", \"MASTER_ADDR\", \"MASTER_PORT\"]\nmissing = [v for v in required if v not in os.environ]\nif missing:\n    raise SystemExit(f\"Launch with torchrun; missing env: {missing}\")","typeGuard":null,"tryCatchPattern":"try:\n    main(args)\nexcept EnvironmentError as e:\n    logging.error(\"%s — relaunch with: torchrun --nproc_per_node=N %s\", e, sys.argv[0])\n    sys.exit(2)","preventionTips":["Always launch multi-GPU scripts via torchrun/torch.distributed.launch.","Single-GPU debugging: use a dedicated single-process script.","Check init_distributed_mode's required flags for this repo."],"tags":["pytorch","distributed-training","ddp","launch","environment"],"backgroundTag":"distributed-not-initialized","analyzedSha":"1ec3fe6f374fc9969973a61f819de25658595afa","analyzedAt":"2026-08-30T09:19:11.901Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}