Lightning-AI/pytorch-lightning · critical · RuntimeError

Cannot re-initialize CUDA in forked subprocess. To use CUDA

Error message

Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method or avoid CUDA initialization in the main process.

What it means

CUDA context cannot be safely carried into a forked child process: if CUDA was initialized in the parent and the process was forked, torch's _is_in_bad_fork detection triggers and Lightning raises this RuntimeError. The fix is to use the 'spawn' start method or avoid touching CUDA before forking. In interactive sessions the kernel must be restarted because CUDA is already initialized.

Source

Thrown at src/lightning/fabric/strategies/launchers/multiprocessing.py:209

def _check_bad_cuda_fork() -> None:
    """Checks whether it is safe to fork and initialize CUDA in the new processes, and raises an exception if not.

    The error message replaces PyTorch's 'Cannot re-initialize CUDA in forked subprocess' with helpful advice for
    Lightning users.

    """
    # Use PyTorch's internal check for bad fork state, which is more accurate than just checking if CUDA
    # is initialized. This allows passive CUDA initialization (e.g., from library imports or device queries)
    # while still catching actual problematic cases where CUDA context was created before forking.
    _is_in_bad_fork = getattr(torch.cuda, "_is_in_bad_fork", None)
    if _is_in_bad_fork is not None and callable(_is_in_bad_fork) and _is_in_bad_fork():
        message = (
            "Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, "
            "you must use the 'spawn' start method or avoid CUDA initialization in the main process."
        )
        if _IS_INTERACTIVE:
            message += " You will have to restart the Python kernel."
        raise RuntimeError(message)

    # Fallback to the old check if _is_in_bad_fork is not available (older PyTorch versions)
    if _is_in_bad_fork is None and torch.cuda.is_initialized():
        message = (
            "Lightning can't create new processes if CUDA is already initialized. Did you manually call"
            " `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any"
            " other way? Please remove any such calls, or change the selected strategy."
        )
        if _IS_INTERACTIVE:
            message += " You will have to restart the Python kernel."
        raise RuntimeError(message)


def _disable_module_memory_sharing(data: Any) -> Any:
    """Disables memory sharing on parameters and buffers of `nn.Module`s contained in the given collection.

    Note: This is only required when running on CPU.

View on GitHub (pinned to 9fed5c27d2)

Solutions

  1. Change the start method to 'spawn' (Fabric default) so children get a fresh CUDA context
  2. Remove any torch.cuda.* calls / .to('cuda') / tensor allocations on GPU before the launcher runs
  3. In notebooks/Jupyter, restart the kernel after removing the CUDA-initializing code, then rerun
  4. If you only fork for CPU dataloaders, keep models and tensors on CPU until after workers start

Example fix

# before
torch.cuda.init()  # or model.to("cuda")
fabric = Fabric(accelerator="gpu", devices=2, strategy="ddp_spawn")

# after
# no CUDA calls before launch; spawn start method keeps a clean context
fabric = Fabric(accelerator="gpu", devices=2, strategy="ddp_spawn")
model = fabric.setup(model)  # device placement happens inside workers
Defensive patterns

Strategy: validation

Validate before calling

import torch
if torch.cuda.is_initialized() and start_method == "fork":
    raise RuntimeError("switch to spawn or de-init CUDA usage before launch")

Try / catch

try:
    fabric.run(train)
except RuntimeError as e:
    if "Cannot re-initialize CUDA in forked subprocess" in str(e):
        # restart kernel / rerun with spawn, after removing CUDA init
        raise
    raise

Prevention

When it happens

Trigger: Calling torch.cuda.* functions (e.g. torch.cuda.is_available() side effects that init CUDA, moving a tensor/model to gpu) before Fabric.fit/run launches forked workers; using start_method='fork' (or the XLA launcher which forks) with CUDA devices; running in a Jupyter notebook where a prior cell initialized CUDA.

Common situations: Setting CUDA_VISIBLE_DEVICES or benchmarking GPU code at the top of a script, then using Fabric with num_workers>1 and fork; notebooks where model.to('cuda') ran in an earlier cell; defaulting to fork on Linux for speed.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/23ad1a28df8350c9. Report an issue: GitHub.