{"record":{"id":"cb45cd4c601c9d51","repo":"WZMIAOMIAO/deep-learning-for-image-processing","slug":"not-find-gpu-device-for-training-cb45cd","errorCode":null,"errorMessage":"not find GPU device for training.","messagePattern":"not find GPU device for training\\.","errorType":"exception","errorClass":"EnvironmentError","httpStatus":null,"severity":"critical","filePath":"pytorch_classification/train_multi_GPU/train_multi_gpu_using_launch.py","lineNumber":21,"sourceCode":"import tempfile\nimport argparse\n\nimport torch\nimport torch.optim as optim\nimport torch.optim.lr_scheduler as lr_scheduler\nfrom torch.utils.tensorboard import SummaryWriter\nfrom torchvision import transforms\n\nfrom model import resnet34\nfrom my_dataset import MyDataSet\nfrom utils import read_split_data, plot_data_loader_image\nfrom multi_train_utils.distributed_utils import init_distributed_mode, dist, cleanup\nfrom multi_train_utils.train_eval_utils import train_one_epoch, evaluate\n\n\ndef main(args):\n    if torch.cuda.is_available() is False:\n        raise EnvironmentError(\"not find GPU device for training.\")\n\n    # 初始化各进程环境\n    init_distributed_mode(args=args)\n\n    rank = args.rank\n    device = torch.device(args.device)\n    batch_size = args.batch_size\n    weights_path = args.weights\n    args.lr *= args.world_size  # 学习率要根据并行GPU的数量进行倍增\n    checkpoint_path = \"\"\n\n    if rank == 0:  # 在第一个进程中打印信息，并实例化tensorboard\n        print(args)\n        print('Start Tensorboard with \"tensorboard --logdir=runs\", view at http://localhost:6006/')\n        tb_writer = SummaryWriter()\n        if os.path.exists(\"./weights\") is False:\n            os.makedirs(\"./weights\")\n","sourceCodeStart":3,"sourceCodeEnd":39,"githubUrl":"https://github.com/WZMIAOMIAO/deep-learning-for-image-processing/blob/1ec3fe6f374fc9969973a61f819de25658595afa/pytorch_classification/train_multi_GPU/train_multi_gpu_using_launch.py#L3-L39","documentation":"This EnvironmentError is raised at the very start of the distributed multi-GPU training entrypoint when torch.cuda.is_available() returns False. The script requires at least one CUDA GPU because it initializes distributed training across GPU processes; without CUDA the entire run is impossible, so the library fails fast with a clear message instead of crashing later inside init_distributed_mode.","triggerScenarios":"Calling python train_multi_gpu_using_launch.py (or torchrun) on a machine where torch.cuda.is_available() is False: no NVIDIA GPU present, PyTorch CPU-only build installed, missing/incompatible NVIDIA driver, or CUDA_VISIBLE_DEVICES set to an empty string.","commonSituations":"Running on a laptop/CI container without GPUs; installing 'pip install torch' which resolves to a CPU wheel on some platforms; driver mismatch after a CUDA toolkit upgrade; running inside a Docker image without --gpus all; setting CUDA_VISIBLE_DEVICES='' to hide GPUs.","solutions":["Verify PyTorch sees the GPU: python -c \"import torch; print(torch.cuda.is_available(), torch.cuda.device_count())\".","If False, install the CUDA build of PyTorch matching your driver, e.g. pip install torch --index-url https://download.pytorch.org/whl/cu121.","Check nvidia-smi works and the driver version supports your CUDA runtime; reinstall the NVIDIA driver if not.","Unset or fix CUDA_VISIBLE_DEVICES so GPUs are visible to the process.","If no GPU exists, run the single-GPU/CPU script (train_single_gpu.py) instead of the multi-GPU launcher."],"exampleFix":"# before\ncuda_visible_devices=\"\" python -m torch.distributed.launch --nproc_per_node=2 train_multi_gpu_using_launch.py\n# after\ncuda_visible_devices=\"0,1\" python -m torch.distributed.launch --nproc_per_node=2 train_multi_gpu_using_launch.py","handlingStrategy":"validation","validationCode":"import torch\nif not torch.cuda.is_available() or torch.cuda.device_count() == 0:\n    raise SystemExit(\"No CUDA GPU visible; install CUDA torch / fix driver before training\")","typeGuard":"def has_cuda(min_count: int = 1) -> bool:\n    import torch\n    return torch.cuda.is_available() and torch.cuda.device_count() >= min_count","tryCatchPattern":"try:\n    main(args)\nexcept EnvironmentError as e:\n    logging.error(\"GPU unavailable: %s — falling back to CPU script\", e)\n    run_single_gpu_or_cpu(args)","preventionTips":["Pin the CUDA-enabled torch wheel in requirements (e.g. --index-url .../whl/cu121).","Run nvidia-smi and the torch.cuda check in CI before training jobs.","Never blanket-set CUDA_VISIBLE_DEVICES in shell profiles.","Document GPU requirement in the script's README."],"tags":["pytorch","cuda","gpu","environment","distributed-training"],"backgroundTag":"no-cuda-device-available","analyzedSha":"1ec3fe6f374fc9969973a61f819de25658595afa","analyzedAt":"2026-08-30T09:19:11.901Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}