ultralytics/ultralytics · critical · RuntimeError

Multi-XPU training requires XCCL, which is not available in

Error message

Multi-XPU training requires XCCL, which is not available in this PyTorch build.

What it means

Raised by BaseTrainer._setup_ddp() when distributed training is launched on multiple Intel XPU (GPU) devices but this PyTorch build lacks the XCCL collective-communication library that multi-XPU DDP requires. The code explicitly checks torch.distributed.is_xccl_available() before selecting the 'xccl' backend for dist.init_process_group(). Without XCCL there is no valid backend for cross-XPU gradient synchronization, so training aborts before the process group forms.

Source

Thrown at ultralytics/engine/trainer.py:270

    def _get_warmup_iterations(self, num_batches):
        """Return warmup iterations, leaving at least the final epoch for regular training."""
        warmup_epochs = min(self.args.warmup_epochs, max(self.epochs - 1, 0))
        return round(warmup_epochs * num_batches) if warmup_epochs > 0 else 0

    def _setup_ddp(self):
        """Initialize and set the DistributedDataParallel parameters for training."""
        device_type = self.args.device.split(":", 1)[0]
        device_type = device_type if device_type in {"npu", "xpu"} else "cuda"
        devices = self.args.device.split(":", 1)[-1].split(",")
        index = int(devices[LOCAL_RANK])  # world_size > 1 guarantees a multi-device string
        self.device = torch.device(device_type, index)
        self.accelerator = get_torch_device_backend(self.device)
        self.accelerator.set_device(index)
        if device_type == "cuda":
            os.environ["TORCH_NCCL_BLOCKING_WAIT"] = "1"  # set to enforce timeout
        elif device_type == "xpu" and not (hasattr(dist, "is_xccl_available") and dist.is_xccl_available()):
            raise RuntimeError("Multi-XPU training requires XCCL, which is not available in this PyTorch build.")
        dist.init_process_group(
            backend={"npu": "hccl", "xpu": "xccl"}.get(device_type, "nccl" if dist.is_nccl_available() else "gloo"),
            timeout=timedelta(seconds=10800),  # 3 hours
            rank=RANK,
            world_size=self.world_size,
        )

    def _build_train_pipeline(self):
        """Build dataloaders, optimizer, and scheduler for current batch size."""
        batch_size = self.batch_size // max(self.world_size, 1)
        self.train_loader = self.get_dataloader(
            self.data["train"], batch_size=batch_size, rank=LOCAL_RANK, mode="train"
        )
        final_batch_size = len(self.train_loader.sampler) % self.train_loader.batch_size or self.train_loader.batch_size
        if self.args.imgsz < 2 * self.stride and not self.train_loader.drop_last and final_batch_size == 1:
            raise ValueError(
                f"final batch=1 training at imgsz={self.args.imgsz} gives BatchNorm a single value per channel; "
                f"change batch or use imgsz >= {2 * self.stride}"

View on GitHub (pinned to 0449ea011c)

Solutions

1. Install a PyTorch build with XCCL support (Intel's XPU-enabled torch wheels, e.g. via intel-extension-for-pytorch and oneCCL bindings), then retry the same command.
2. Verify availability first in Python: import torch.distributed as dist; print(hasattr(dist, 'is_xccl_available') and dist.is_xccl_available()).
3. If XCCL cannot be installed, fall back to single-XPU training: `yolo train device=xpu:0`.
4. If you intended CUDA training, check that the device string was not mistyped as 'xpu' when 'cuda' was meant.

Example fix

# before
yolo train device=xpu:0,1  # RuntimeError: Multi-XPU training requires XCCL

# after
# single XPU, no collectives needed
yolo train device=xpu:0

import torch.distributed as dist

def can_multi_xpu():
    return hasattr(dist, "is_xccl_available") and dist.is_xccl_available()

if not can_multi_xpu():
    device = "xpu:0"  # fall back to single XPU instead of launching DDP

try:
    trainer = Model(...).train(device="xpu:0,1", ...)
except RuntimeError as e:
    if "XCCL" in str(e):
        log.warning("XCCL missing; rerunning on single XPU")
        Model(...).train(device="xpu:0", ...)
    else:
        raise

When it happens

Trigger: Running a command like `yolo train device=xpu:0,1` (or torchrun with >1 XPU rank) on a PyTorch wheel compiled without oneAPI/XCCL support. The xpu path in _setup_ddp is only reached when world_size > 1 and the device string starts with 'xpu'.

Common situations: Using a stock upstream PyTorch wheel (pip install torch) on Intel GPUs instead of the Intel-extended intel-extension-for-pytorch / oneCCL builds; upgrading PyTorch to a version that dropped XCCL; CI runners with XPU hardware but a default torch install.

Related errors

• '{k}={v}' is of invalid type {type(v).__name__}. Valid '{k}'
• 'source=0' webcam not supported in Colab and Kaggle notebook
• AutoBatch with batch<1 not supported for Multi-GPU training,
• Ascend inference requires the CANN runtime and `ais_bench` P
• DEEPX inference requires the DEEPX DX-Runtime and `dx_engine

AI-assisted analysis of ultralytics/ultralytics@0449ea011c (2026-08-15). Data as JSON: /api/errors/03b54d24f4e76174. Report an issue: GitHub.