{"record":{"id":"5815c665fea6414d","repo":"WZMIAOMIAO/deep-learning-for-image-processing","slug":"not-find-gpu-device-for-training-5815c6","errorCode":null,"errorMessage":"not find GPU device for training.","messagePattern":"not find GPU device for training\\.","errorType":"exception","errorClass":"EnvironmentError","httpStatus":null,"severity":"critical","filePath":"pytorch_classification/train_multi_GPU/train_multi_gpu_using_spawn.py","lineNumber":23,"sourceCode":"\nimport torch\nimport torch.multiprocessing as mp\nfrom torch.multiprocessing import Process\nimport torch.optim as optim\nimport torch.optim.lr_scheduler as lr_scheduler\nfrom torch.utils.tensorboard import SummaryWriter\nfrom torchvision import transforms\n\nfrom model import resnet34\nfrom my_dataset import MyDataSet\nfrom utils import read_split_data, plot_data_loader_image\nfrom multi_train_utils.distributed_utils import dist, cleanup\nfrom multi_train_utils.train_eval_utils import train_one_epoch, evaluate\n\n\ndef main_fun(rank, world_size, args):\n    if torch.cuda.is_available() is False:\n        raise EnvironmentError(\"not find GPU device for training.\")\n\n    # 初始化各进程环境 start\n    os.environ[\"MASTER_ADDR\"] = \"localhost\"\n    os.environ[\"MASTER_PORT\"] = \"12355\"\n\n    args.rank = rank\n    args.world_size = world_size\n    args.gpu = rank\n\n    args.distributed = True\n\n    torch.cuda.set_device(args.gpu)\n    args.dist_backend = 'nccl'\n    print('| distributed init (rank {}): {}'.format(\n        args.rank, args.dist_url), flush=True)\n    dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,\n                            world_size=args.world_size, rank=args.rank)\n    dist.barrier()","sourceCodeStart":5,"sourceCodeEnd":41,"githubUrl":"https://github.com/WZMIAOMIAO/deep-learning-for-image-processing/blob/1ec3fe6f374fc9969973a61f819de25658595afa/pytorch_classification/train_multi_GPU/train_multi_gpu_using_spawn.py#L5-L41","documentation":"Identical guard to the launch-based variant, but in the torch.multiprocessing.spawn version of multi-GPU training. Each spawned process runs main_fun(rank, world_size, args), and the first statement checks torch.cuda.is_available(); if False it raises EnvironmentError because spawn-based DDP requires CUDA devices to place one process per GPU.","triggerScenarios":"Running the spawn-based trainer on a machine with no usable CUDA: GPU-less host, CPU-only torch wheel, broken driver, or CUDA_VISIBLE_DEVICES hiding all devices — the check fires in every spawned worker process.","commonSituations":"Wrong torch build (pip picked the +cpu wheel); WSL/Docker without GPU passthrough; driver/libcuda mismatch inside the container; forgetting that spawn re-imports the module in each worker so a bad env repeats the error per process.","solutions":["Run python -c \"import torch; print(torch.cuda.is_available())\" to confirm CUDA visibility.","Reinstall the CUDA-enabled PyTorch wheel matching your CUDA driver (see pytorch.org install matrix).","Fix the NVIDIA driver / container GPU flags (docker run --gpus all).","Ensure CUDA_VISIBLE_DEVICES lists available GPUs before spawning.","Fall back to single-GPU or CPU training if the machine genuinely has no GPU."],"exampleFix":"# before\nif torch.cuda.is_available() is False:\n    raise EnvironmentError(\"not find GPU device for training.\")\n# after (caller-side precheck)\nimport torch\nassert torch.cuda.is_available() and torch.cuda.device_count() >= args.world_size, \"need CUDA GPUs\"\nmp.spawn(main_fun, args=(args.world_size, args), nprocs=args.world_size)","handlingStrategy":"validation","validationCode":"import torch, os\nassert torch.cuda.is_available(), \"CUDA unavailable\"\nassert torch.cuda.device_count() >= world_size, f\"need {world_size} GPUs, found {torch.cuda.device_count()}\"","typeGuard":"def gpus_ready(n: int) -> bool:\n    import torch\n    return torch.cuda.is_available() and torch.cuda.device_count() >= n","tryCatchPattern":"try:\n    mp.spawn(main_fun, args=(world_size, args), nprocs=world_size)\nexcept EnvironmentError as e:\n    logging.error(\"spawn aborted: %s\", e)\n    sys.exit(2)","preventionTips":["Set MASTER_ADDR/MASTER_PORT before spawn.","Use torchrun instead of manual mp.spawn where possible.","Verify torch.cuda.device_count() matches --nproc_per_node."],"tags":["pytorch","cuda","gpu","multiprocessing","distributed-training"],"backgroundTag":"no-cuda-device-available","analyzedSha":"1ec3fe6f374fc9969973a61f819de25658595afa","analyzedAt":"2026-08-30T09:19:11.901Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}