Lightning-AI/pytorch-lightning · critical · RuntimeError

Lightning can't create new processes if CUDA is already init

Error message

Lightning can't create new processes if CUDA is already initialized. Did you manually call `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any other way? Please remove any such calls, or change the selected strategy.

What it means

This is the fallback check for older PyTorch versions that lack _is_in_bad_fork: if torch.cuda.is_initialized() is True in the parent process, Lightning refuses to create child processes because forking with a live CUDA context leads to crashes or corruption. It tells you to remove any CUDA initialization before launch or change strategy.

Source

Thrown at src/lightning/fabric/strategies/launchers/multiprocessing.py:220

    if _is_in_bad_fork is not None and callable(_is_in_bad_fork) and _is_in_bad_fork():
        message = (
            "Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, "
            "you must use the 'spawn' start method or avoid CUDA initialization in the main process."
        )
        if _IS_INTERACTIVE:
            message += " You will have to restart the Python kernel."
        raise RuntimeError(message)

    # Fallback to the old check if _is_in_bad_fork is not available (older PyTorch versions)
    if _is_in_bad_fork is None and torch.cuda.is_initialized():
        message = (
            "Lightning can't create new processes if CUDA is already initialized. Did you manually call"
            " `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any"
            " other way? Please remove any such calls, or change the selected strategy."
        )
        if _IS_INTERACTIVE:
            message += " You will have to restart the Python kernel."
        raise RuntimeError(message)


def _disable_module_memory_sharing(data: Any) -> Any:
    """Disables memory sharing on parameters and buffers of `nn.Module`s contained in the given collection.

    Note: This is only required when running on CPU.

    """
    # PyTorch enables memory sharing automatically on all tensors that are passed through `mp.spawn`.
    # For model weights and buffers, this is undesired and can lead to race conditions between processes.
    # Hence, we copy the tensors in the entire module to ensure it doesn't share memory with other processes.

    @torch.no_grad()
    def unshare(module: Module) -> Module:
        for tensor in itertools.chain(module.parameters(), module.buffers()):
            tensor.data = tensor.data.clone()
        return module

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Remove or move all CUDA-touching code until after worker processes are launched
2. Use a strategy that doesn't fork/spawn from the initialized process, e.g. subprocess-script based 'ddp' launcher
3. Upgrade PyTorch so the precise _is_in_bad_fork check is used
4. Restart the kernel/session to clear the initialized CUDA state, then rerun without the offending calls

Example fix

# before
x = torch.randn(4, device="cuda")  # initializes CUDA in parent
fabric = Fabric(accelerator="gpu", devices=4)
fabric.run(train)

# after
# keep parent process CUDA-free
fabric = Fabric(accelerator="gpu", devices=4)
fabric.run(train)  # allocate CUDA tensors inside train()

import torch
if torch.cuda.is_initialized():
    raise SystemExit("CUDA already initialized before launching workers; move CUDA usage into workers")

When it happens

Trigger: Any torch.cuda.* call, GPU tensor allocation, or model.to('cuda') in the main process before the multiprocessing launcher runs, on a PyTorch version without _is_in_bad_fork.

Common situations: Warming up the GPU, printing torch.cuda.get_device_name(), setting cudnn benchmark flags, or moving data to GPU for a quick test before calling Fabric.run; older pinned PyTorch versions in Docker images.

Related errors

• Cannot re-initialize CUDA in forked subprocess. To use CUDA
• The start method '{self._start_method}' is not available on
• Launching multiple processes with the 'spawn' start method r
• GPUs requested but none are available.
• The start method '{self._start_method}' is not available on

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/31174ca1cae85cc1. Report an issue: GitHub.