WZMIAOMIAO/deep-learning-for-image-processing · critical · EnvironmentError

not find GPU device for training.

Error message

not find GPU device for training.

What it means

This EnvironmentError is raised at the very start of the distributed multi-GPU training entrypoint when torch.cuda.is_available() returns False. The script requires at least one CUDA GPU because it initializes distributed training across GPU processes; without CUDA the entire run is impossible, so the library fails fast with a clear message instead of crashing later inside init_distributed_mode.

Source

Thrown at pytorch_classification/train_multi_GPU/train_multi_gpu_using_launch.py:21

import tempfile
import argparse

import torch
import torch.optim as optim
import torch.optim.lr_scheduler as lr_scheduler
from torch.utils.tensorboard import SummaryWriter
from torchvision import transforms

from model import resnet34
from my_dataset import MyDataSet
from utils import read_split_data, plot_data_loader_image
from multi_train_utils.distributed_utils import init_distributed_mode, dist, cleanup
from multi_train_utils.train_eval_utils import train_one_epoch, evaluate


def main(args):
    if torch.cuda.is_available() is False:
        raise EnvironmentError("not find GPU device for training.")

    # 初始化各进程环境
    init_distributed_mode(args=args)

    rank = args.rank
    device = torch.device(args.device)
    batch_size = args.batch_size
    weights_path = args.weights
    args.lr *= args.world_size  # 学习率要根据并行GPU的数量进行倍增
    checkpoint_path = ""

    if rank == 0:  # 在第一个进程中打印信息，并实例化tensorboard
        print(args)
        print('Start Tensorboard with "tensorboard --logdir=runs", view at http://localhost:6006/')
        tb_writer = SummaryWriter()
        if os.path.exists("./weights") is False:
            os.makedirs("./weights")

View on GitHub (pinned to 1ec3fe6f37)

Solutions

1. Verify PyTorch sees the GPU: python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())".
2. If False, install the CUDA build of PyTorch matching your driver, e.g. pip install torch --index-url https://download.pytorch.org/whl/cu121.
3. Check nvidia-smi works and the driver version supports your CUDA runtime; reinstall the NVIDIA driver if not.
4. Unset or fix CUDA_VISIBLE_DEVICES so GPUs are visible to the process.
5. If no GPU exists, run the single-GPU/CPU script (train_single_gpu.py) instead of the multi-GPU launcher.

Example fix

# before
cuda_visible_devices="" python -m torch.distributed.launch --nproc_per_node=2 train_multi_gpu_using_launch.py
# after
cuda_visible_devices="0,1" python -m torch.distributed.launch --nproc_per_node=2 train_multi_gpu_using_launch.py

import torch
if not torch.cuda.is_available() or torch.cuda.device_count() == 0:
    raise SystemExit("No CUDA GPU visible; install CUDA torch / fix driver before training")

def has_cuda(min_count: int = 1) -> bool:
    import torch
    return torch.cuda.is_available() and torch.cuda.device_count() >= min_count

try:
    main(args)
except EnvironmentError as e:
    logging.error("GPU unavailable: %s — falling back to CPU script", e)
    run_single_gpu_or_cpu(args)

When it happens

Trigger: Calling python train_multi_gpu_using_launch.py (or torchrun) on a machine where torch.cuda.is_available() is False: no NVIDIA GPU present, PyTorch CPU-only build installed, missing/incompatible NVIDIA driver, or CUDA_VISIBLE_DEVICES set to an empty string.

Common situations: Running on a laptop/CI container without GPUs; installing 'pip install torch' which resolves to a CPU wheel on some platforms; driver mismatch after a CUDA toolkit upgrade; running inside a Docker image without --gpus all; setting CUDA_VISIBLE_DEVICES='' to hide GPUs.

Related errors

• not find GPU device for training.
• not find GPU device for training.
• not find GPU device for training.
• not support distributed training.
• not support data format '{self.data_format}'

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/cb45cd4c601c9d51. Report an issue: GitHub.