{"record":{"id":"31174ca1cae85cc1","repo":"Lightning-AI/pytorch-lightning","slug":"lightning-can-t-create-new-processes-if-cuda-is-al","errorCode":null,"errorMessage":"Lightning can't create new processes if CUDA is already initialized. Did you manually call `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any other way? Please remove any such calls, or change the selected strategy.","messagePattern":"Lightning can't create new processes if CUDA is already initialized\\. Did you manually call `torch\\.cuda\\.\\*` functions, have moved the model to the device, or allocated memory on the GPU any other way\\? Please remove any such calls, or change the selected strategy\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/lightning/fabric/strategies/launchers/multiprocessing.py","lineNumber":220,"sourceCode":"    if _is_in_bad_fork is not None and callable(_is_in_bad_fork) and _is_in_bad_fork():\n        message = (\n            \"Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, \"\n            \"you must use the 'spawn' start method or avoid CUDA initialization in the main process.\"\n        )\n        if _IS_INTERACTIVE:\n            message += \" You will have to restart the Python kernel.\"\n        raise RuntimeError(message)\n\n    # Fallback to the old check if _is_in_bad_fork is not available (older PyTorch versions)\n    if _is_in_bad_fork is None and torch.cuda.is_initialized():\n        message = (\n            \"Lightning can't create new processes if CUDA is already initialized. Did you manually call\"\n            \" `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any\"\n            \" other way? Please remove any such calls, or change the selected strategy.\"\n        )\n        if _IS_INTERACTIVE:\n            message += \" You will have to restart the Python kernel.\"\n        raise RuntimeError(message)\n\n\ndef _disable_module_memory_sharing(data: Any) -> Any:\n    \"\"\"Disables memory sharing on parameters and buffers of `nn.Module`s contained in the given collection.\n\n    Note: This is only required when running on CPU.\n\n    \"\"\"\n    # PyTorch enables memory sharing automatically on all tensors that are passed through `mp.spawn`.\n    # For model weights and buffers, this is undesired and can lead to race conditions between processes.\n    # Hence, we copy the tensors in the entire module to ensure it doesn't share memory with other processes.\n\n    @torch.no_grad()\n    def unshare(module: Module) -> Module:\n        for tensor in itertools.chain(module.parameters(), module.buffers()):\n            tensor.data = tensor.data.clone()\n        return module\n","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/launchers/multiprocessing.py#L202-L238","documentation":"This is the fallback check for older PyTorch versions that lack _is_in_bad_fork: if torch.cuda.is_initialized() is True in the parent process, Lightning refuses to create child processes because forking with a live CUDA context leads to crashes or corruption. It tells you to remove any CUDA initialization before launch or change strategy.","triggerScenarios":"Any torch.cuda.* call, GPU tensor allocation, or model.to('cuda') in the main process before the multiprocessing launcher runs, on a PyTorch version without _is_in_bad_fork.","commonSituations":"Warming up the GPU, printing torch.cuda.get_device_name(), setting cudnn benchmark flags, or moving data to GPU for a quick test before calling Fabric.run; older pinned PyTorch versions in Docker images.","solutions":["Remove or move all CUDA-touching code until after worker processes are launched","Use a strategy that doesn't fork/spawn from the initialized process, e.g. subprocess-script based 'ddp' launcher","Upgrade PyTorch so the precise _is_in_bad_fork check is used","Restart the kernel/session to clear the initialized CUDA state, then rerun without the offending calls"],"exampleFix":"# before\nx = torch.randn(4, device=\"cuda\")  # initializes CUDA in parent\nfabric = Fabric(accelerator=\"gpu\", devices=4)\nfabric.run(train)\n\n# after\n# keep parent process CUDA-free\nfabric = Fabric(accelerator=\"gpu\", devices=4)\nfabric.run(train)  # allocate CUDA tensors inside train()","handlingStrategy":"validation","validationCode":"import torch\nif torch.cuda.is_initialized():\n    raise SystemExit(\"CUDA already initialized before launching workers; move CUDA usage into workers\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Defer device placement to inside the launched function","Pin recent PyTorch so _is_in_bad_fork gives precise detection","Put GPU warmup/benchmarking in a separate script or after workers start"],"tags":["pytorch-lightning","cuda","multiprocessing","pytorch-version"],"backgroundTag":"cuda-initialized-before-fork","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}