WZMIAOMIAO/deep-learning-for-image-processing · critical · EnvironmentError

not find GPU device for training.

Error message

not find GPU device for training.

What it means

Identical guard to the launch-based variant, but in the torch.multiprocessing.spawn version of multi-GPU training. Each spawned process runs main_fun(rank, world_size, args), and the first statement checks torch.cuda.is_available(); if False it raises EnvironmentError because spawn-based DDP requires CUDA devices to place one process per GPU.

Source

Thrown at pytorch_classification/train_multi_GPU/train_multi_gpu_using_spawn.py:23

import torch
import torch.multiprocessing as mp
from torch.multiprocessing import Process
import torch.optim as optim
import torch.optim.lr_scheduler as lr_scheduler
from torch.utils.tensorboard import SummaryWriter
from torchvision import transforms

from model import resnet34
from my_dataset import MyDataSet
from utils import read_split_data, plot_data_loader_image
from multi_train_utils.distributed_utils import dist, cleanup
from multi_train_utils.train_eval_utils import train_one_epoch, evaluate


def main_fun(rank, world_size, args):
    if torch.cuda.is_available() is False:
        raise EnvironmentError("not find GPU device for training.")

    # 初始化各进程环境 start
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355"

    args.rank = rank
    args.world_size = world_size
    args.gpu = rank

    args.distributed = True

    torch.cuda.set_device(args.gpu)
    args.dist_backend = 'nccl'
    print('| distributed init (rank {}): {}'.format(
        args.rank, args.dist_url), flush=True)
    dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,
                            world_size=args.world_size, rank=args.rank)
    dist.barrier()

View on GitHub (pinned to 1ec3fe6f37)

Solutions

1. Run python -c "import torch; print(torch.cuda.is_available())" to confirm CUDA visibility.
2. Reinstall the CUDA-enabled PyTorch wheel matching your CUDA driver (see pytorch.org install matrix).
3. Fix the NVIDIA driver / container GPU flags (docker run --gpus all).
4. Ensure CUDA_VISIBLE_DEVICES lists available GPUs before spawning.
5. Fall back to single-GPU or CPU training if the machine genuinely has no GPU.

Example fix

# before
if torch.cuda.is_available() is False:
    raise EnvironmentError("not find GPU device for training.")
# after (caller-side precheck)
import torch
assert torch.cuda.is_available() and torch.cuda.device_count() >= args.world_size, "need CUDA GPUs"
mp.spawn(main_fun, args=(args.world_size, args), nprocs=args.world_size)

import torch, os
assert torch.cuda.is_available(), "CUDA unavailable"
assert torch.cuda.device_count() >= world_size, f"need {world_size} GPUs, found {torch.cuda.device_count()}"

def gpus_ready(n: int) -> bool:
    import torch
    return torch.cuda.is_available() and torch.cuda.device_count() >= n

try:
    mp.spawn(main_fun, args=(world_size, args), nprocs=world_size)
except EnvironmentError as e:
    logging.error("spawn aborted: %s", e)
    sys.exit(2)

When it happens

Trigger: Running the spawn-based trainer on a machine with no usable CUDA: GPU-less host, CPU-only torch wheel, broken driver, or CUDA_VISIBLE_DEVICES hiding all devices — the check fires in every spawned worker process.

Common situations: Wrong torch build (pip picked the +cpu wheel); WSL/Docker without GPU passthrough; driver/libcuda mismatch inside the container; forgetting that spawn re-imports the module in each worker so a bad env repeats the error per process.

Related errors

• not find GPU device for training.
• not find GPU device for training.
• not find GPU device for training.
• not support distributed training.
• not support data format '{self.data_format}'

AI-assisted analysis of WZMIAOMIAO/deep-learning-for-image-processing@1ec3fe6f37 (2026-08-30). Data as JSON: /api/errors/5815c665fea6414d. Report an issue: GitHub.