{"record":{"id":"38e408e8be6df0c0","repo":"huggingface/transformers","slug":"we-tried-to-initialize-torch-distributed-for-you","errorCode":null,"errorMessage":"We tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use distributed training.","messagePattern":"We tried to initialize torch\\.distributed for you, but it failed\\. Make sure you init torch distributed in your script to use distributed training\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"src/transformers/distributed/utils.py","lineNumber":94,"sourceCode":"                \"xpu\": \"xccl\",\n                \"hpu\": \"hccl\",\n                \"neuron\": \"neuron\",\n                \"tpu\": \"tpu_dist\",\n            }\n            backend = backend_map.get(device_type)\n\n            # Bind the accelerator before init so the process group is created with a\n            # device_id, otherwise collectives like barrier() warn (and may spin up an\n            # extra NCCL comm) about the missing device binding.\n            device_id = None\n            if device_type != \"cpu\":\n                getattr(torch, device_type).set_device(local_rank)\n                device_id = torch.device(device_type, local_rank)\n            torch.distributed.init_process_group(\n                backend=backend, rank=rank, world_size=world_size, device_id=device_id\n            )\n        except Exception as e:\n            raise OSError(\n                \"We tried to initialize torch.distributed for you, but it failed. Make \"\n                \"sure you init torch distributed in your script to use distributed training.\"\n            ) from e\n\n\ndef _distributed_barrier():\n    \"\"\"Barrier bound to the current accelerator device.\n\n    Passing `device_ids` is required when the process group was initialized without a\n    `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call\n    when torch.distributed has not been initialized — returns immediately.\n    \"\"\"\n    if not _is_torch_distributed_initialized():\n        return\n    device_type = torch._C._get_accelerator().type\n    if device_type != \"cpu\":\n        torch.distributed.barrier(device_ids=[getattr(torch, device_type).current_device()])\n    else:","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/distributed/utils.py#L76-L112","documentation":"When distributed training is requested but torch.distributed is not initialized, transformers attempts an automatic init_process_group (binding the accelerator device for NCCL). This OSError means that auto-init itself raised — the wrapped exception ('from e') holds the real cause, typically missing/invalid rendezvous environment variables or a backend failure.","triggerScenarios":"Running a script that requests tp_size/fsdp_size > 1 with plain python (no RANK/WORLD_SIZE/MASTER_ADDR set), wrong LOCAL_RANK for the available GPUs, an unavailable NCCL backend, or port conflicts on the rendezvous address.","commonSituations":"Running torchrun-launched configs from an IDE or notebook; stale MASTER_PORT from a crashed job; NCCL not installed/visible in a container; GPU driver/device mismatches.","solutions":["Inspect the chained cause (raise ... from e) — the original exception names the actual problem.","Launch via torchrun --nproc_per_node=N script.py so all env vars are set correctly.","If you must init yourself, call torch.distributed.init_process_group(backend='nccl') at script start before loading the model with a distributed_config.","Check MASTER_PORT availability and NCCL installation for GPU runs."],"exampleFix":"# before\npython train.py --tp_size 2  # no rendezvous env, auto-init fails\n\n# after\ntorchrun --nproc_per_node=2 train.py --tp_size 2","handlingStrategy":"try-catch","validationCode":"import os, torch.distributed as dist\n\ndef env_ready_for_dist() -> bool:\n    return all(k in os.environ for k in (\"RANK\", \"WORLD_SIZE\", \"MASTER_ADDR\", \"MASTER_PORT\")) and dist.is_available()","typeGuard":null,"tryCatchPattern":"try:\n    Model.from_pretrained(model_id, distributed_config=cfg)\nexcept OSError as e:\n    cause = e.__cause__\n    if env_ready_for_dist() and not dist.is_initialized():\n        dist.init_process_group(backend=\"nccl\")\n        Model.from_pretrained(model_id, distributed_config=cfg)\n    else:\n        raise  # inspect e.__cause__ for the real reason","preventionTips":["Always launch distributed training with torchrun.","Init the process group yourself at script start rather than relying on auto-init.","Read the chained cause exception; the wrapper message alone is generic."],"tags":["distributed","process-group","environment","launch"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}