huggingface/transformers · error · OSError

We tried to initialize torch.distributed for you, but it fai

Error message

We tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use distributed training.

What it means

When distributed training is requested but torch.distributed is not initialized, transformers attempts an automatic init_process_group (binding the accelerator device for NCCL). This OSError means that auto-init itself raised — the wrapped exception ('from e') holds the real cause, typically missing/invalid rendezvous environment variables or a backend failure.

Source

Thrown at src/transformers/distributed/utils.py:94

                "xpu": "xccl",
                "hpu": "hccl",
                "neuron": "neuron",
                "tpu": "tpu_dist",
            }
            backend = backend_map.get(device_type)

            # Bind the accelerator before init so the process group is created with a
            # device_id, otherwise collectives like barrier() warn (and may spin up an
            # extra NCCL comm) about the missing device binding.
            device_id = None
            if device_type != "cpu":
                getattr(torch, device_type).set_device(local_rank)
                device_id = torch.device(device_type, local_rank)
            torch.distributed.init_process_group(
                backend=backend, rank=rank, world_size=world_size, device_id=device_id
            )
        except Exception as e:
            raise OSError(
                "We tried to initialize torch.distributed for you, but it failed. Make "
                "sure you init torch distributed in your script to use distributed training."
            ) from e


def _distributed_barrier():
    """Barrier bound to the current accelerator device.

    Passing `device_ids` is required when the process group was initialized without a
    `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call
    when torch.distributed has not been initialized — returns immediately.
    """
    if not _is_torch_distributed_initialized():
        return
    device_type = torch._C._get_accelerator().type
    if device_type != "cpu":
        torch.distributed.barrier(device_ids=[getattr(torch, device_type).current_device()])
    else:

View on GitHub (pinned to a597f97485)

Solutions

Inspect the chained cause (raise ... from e) — the original exception names the actual problem.
Launch via torchrun --nproc_per_node=N script.py so all env vars are set correctly.
If you must init yourself, call torch.distributed.init_process_group(backend='nccl') at script start before loading the model with a distributed_config.
Check MASTER_PORT availability and NCCL installation for GPU runs.

Example fix

# before
python train.py --tp_size 2  # no rendezvous env, auto-init fails

# after
torchrun --nproc_per_node=2 train.py --tp_size 2

Defensive patterns

Strategy: try-catch

Validate before calling

import os, torch.distributed as dist

def env_ready_for_dist() -> bool:
    return all(k in os.environ for k in ("RANK", "WORLD_SIZE", "MASTER_ADDR", "MASTER_PORT")) and dist.is_available()

Try / catch

try:
    Model.from_pretrained(model_id, distributed_config=cfg)
except OSError as e:
    cause = e.__cause__
    if env_ready_for_dist() and not dist.is_initialized():
        dist.init_process_group(backend="nccl")
        Model.from_pretrained(model_id, distributed_config=cfg)
    else:
        raise  # inspect e.__cause__ for the real reason

Prevention

Always launch distributed training with torchrun.
Init the process group yourself at script start rather than relying on auto-init.
Read the chained cause exception; the wrapper message alone is generic.

When it happens

Trigger: Running a script that requests tp_size/fsdp_size > 1 with plain python (no RANK/WORLD_SIZE/MASTER_ADDR set), wrong LOCAL_RANK for the available GPUs, an unavailable NCCL backend, or port conflicts on the rendezvous address.

Common situations: Running torchrun-launched configs from an IDE or notebook; stale MASTER_PORT from a crashed job; NCCL not installed/visible in a container; GPU driver/device mismatches.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/38e408e8be6df0c0. Report an issue: GitHub.