{"record":{"id":"23ad1a28df8350c9","repo":"Lightning-AI/pytorch-lightning","slug":"cannot-re-initialize-cuda-in-forked-subprocess-to","errorCode":null,"errorMessage":"Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method or avoid CUDA initialization in the main process.","messagePattern":"Cannot re-initialize CUDA in forked subprocess\\. To use CUDA with multiprocessing, you must use the 'spawn' start method or avoid CUDA initialization in the main process\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"src/lightning/fabric/strategies/launchers/multiprocessing.py","lineNumber":209,"sourceCode":"def _check_bad_cuda_fork() -> None:\n    \"\"\"Checks whether it is safe to fork and initialize CUDA in the new processes, and raises an exception if not.\n\n    The error message replaces PyTorch's 'Cannot re-initialize CUDA in forked subprocess' with helpful advice for\n    Lightning users.\n\n    \"\"\"\n    # Use PyTorch's internal check for bad fork state, which is more accurate than just checking if CUDA\n    # is initialized. This allows passive CUDA initialization (e.g., from library imports or device queries)\n    # while still catching actual problematic cases where CUDA context was created before forking.\n    _is_in_bad_fork = getattr(torch.cuda, \"_is_in_bad_fork\", None)\n    if _is_in_bad_fork is not None and callable(_is_in_bad_fork) and _is_in_bad_fork():\n        message = (\n            \"Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, \"\n            \"you must use the 'spawn' start method or avoid CUDA initialization in the main process.\"\n        )\n        if _IS_INTERACTIVE:\n            message += \" You will have to restart the Python kernel.\"\n        raise RuntimeError(message)\n\n    # Fallback to the old check if _is_in_bad_fork is not available (older PyTorch versions)\n    if _is_in_bad_fork is None and torch.cuda.is_initialized():\n        message = (\n            \"Lightning can't create new processes if CUDA is already initialized. Did you manually call\"\n            \" `torch.cuda.*` functions, have moved the model to the device, or allocated memory on the GPU any\"\n            \" other way? Please remove any such calls, or change the selected strategy.\"\n        )\n        if _IS_INTERACTIVE:\n            message += \" You will have to restart the Python kernel.\"\n        raise RuntimeError(message)\n\n\ndef _disable_module_memory_sharing(data: Any) -> Any:\n    \"\"\"Disables memory sharing on parameters and buffers of `nn.Module`s contained in the given collection.\n\n    Note: This is only required when running on CPU.\n","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/launchers/multiprocessing.py#L191-L227","documentation":"CUDA context cannot be safely carried into a forked child process: if CUDA was initialized in the parent and the process was forked, torch's _is_in_bad_fork detection triggers and Lightning raises this RuntimeError. The fix is to use the 'spawn' start method or avoid touching CUDA before forking. In interactive sessions the kernel must be restarted because CUDA is already initialized.","triggerScenarios":"Calling torch.cuda.* functions (e.g. torch.cuda.is_available() side effects that init CUDA, moving a tensor/model to gpu) before Fabric.fit/run launches forked workers; using start_method='fork' (or the XLA launcher which forks) with CUDA devices; running in a Jupyter notebook where a prior cell initialized CUDA.","commonSituations":"Setting CUDA_VISIBLE_DEVICES or benchmarking GPU code at the top of a script, then using Fabric with num_workers>1 and fork; notebooks where model.to('cuda') ran in an earlier cell; defaulting to fork on Linux for speed.","solutions":["Change the start method to 'spawn' (Fabric default) so children get a fresh CUDA context","Remove any torch.cuda.* calls / .to('cuda') / tensor allocations on GPU before the launcher runs","In notebooks/Jupyter, restart the kernel after removing the CUDA-initializing code, then rerun","If you only fork for CPU dataloaders, keep models and tensors on CPU until after workers start"],"exampleFix":"# before\ntorch.cuda.init()  # or model.to(\"cuda\")\nfabric = Fabric(accelerator=\"gpu\", devices=2, strategy=\"ddp_spawn\")\n\n# after\n# no CUDA calls before launch; spawn start method keeps a clean context\nfabric = Fabric(accelerator=\"gpu\", devices=2, strategy=\"ddp_spawn\")\nmodel = fabric.setup(model)  # device placement happens inside workers","handlingStrategy":"validation","validationCode":"import torch\nif torch.cuda.is_initialized() and start_method == \"fork\":\n    raise RuntimeError(\"switch to spawn or de-init CUDA usage before launch\")","typeGuard":null,"tryCatchPattern":"try:\n    fabric.run(train)\nexcept RuntimeError as e:\n    if \"Cannot re-initialize CUDA in forked subprocess\" in str(e):\n        # restart kernel / rerun with spawn, after removing CUDA init\n        raise\n    raise","preventionTips":["Keep the parent process CUDA-free: no torch.cuda.* calls, no .to('cuda') before launch","Use the default spawn start method for GPU multi-process training","In notebooks, restart the kernel after debugging GPU code before launching distributed runs"],"tags":["pytorch-lightning","cuda","multiprocessing","fork","spawn"],"backgroundTag":"cuda-fork-reinitialization","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}