{"record":{"id":"03b54d24f4e76174","repo":"ultralytics/ultralytics","slug":"multi-xpu-training-requires-xccl-which-is-not-ava","errorCode":null,"errorMessage":"Multi-XPU training requires XCCL, which is not available in this PyTorch build.","messagePattern":"Multi-XPU training requires XCCL, which is not available in this PyTorch build\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"ultralytics/engine/trainer.py","lineNumber":270,"sourceCode":"\n    def _get_warmup_iterations(self, num_batches):\n        \"\"\"Return warmup iterations, leaving at least the final epoch for regular training.\"\"\"\n        warmup_epochs = min(self.args.warmup_epochs, max(self.epochs - 1, 0))\n        return round(warmup_epochs * num_batches) if warmup_epochs > 0 else 0\n\n    def _setup_ddp(self):\n        \"\"\"Initialize and set the DistributedDataParallel parameters for training.\"\"\"\n        device_type = self.args.device.split(\":\", 1)[0]\n        device_type = device_type if device_type in {\"npu\", \"xpu\"} else \"cuda\"\n        devices = self.args.device.split(\":\", 1)[-1].split(\",\")\n        index = int(devices[LOCAL_RANK])  # world_size > 1 guarantees a multi-device string\n        self.device = torch.device(device_type, index)\n        self.accelerator = get_torch_device_backend(self.device)\n        self.accelerator.set_device(index)\n        if device_type == \"cuda\":\n            os.environ[\"TORCH_NCCL_BLOCKING_WAIT\"] = \"1\"  # set to enforce timeout\n        elif device_type == \"xpu\" and not (hasattr(dist, \"is_xccl_available\") and dist.is_xccl_available()):\n            raise RuntimeError(\"Multi-XPU training requires XCCL, which is not available in this PyTorch build.\")\n        dist.init_process_group(\n            backend={\"npu\": \"hccl\", \"xpu\": \"xccl\"}.get(device_type, \"nccl\" if dist.is_nccl_available() else \"gloo\"),\n            timeout=timedelta(seconds=10800),  # 3 hours\n            rank=RANK,\n            world_size=self.world_size,\n        )\n\n    def _build_train_pipeline(self):\n        \"\"\"Build dataloaders, optimizer, and scheduler for current batch size.\"\"\"\n        batch_size = self.batch_size // max(self.world_size, 1)\n        self.train_loader = self.get_dataloader(\n            self.data[\"train\"], batch_size=batch_size, rank=LOCAL_RANK, mode=\"train\"\n        )\n        final_batch_size = len(self.train_loader.sampler) % self.train_loader.batch_size or self.train_loader.batch_size\n        if self.args.imgsz < 2 * self.stride and not self.train_loader.drop_last and final_batch_size == 1:\n            raise ValueError(\n                f\"final batch=1 training at imgsz={self.args.imgsz} gives BatchNorm a single value per channel; \"\n                f\"change batch or use imgsz >= {2 * self.stride}\"","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/ultralytics/ultralytics/blob/0449ea011cfd6c9a0d50a0bf1043aca5190cd476/ultralytics/engine/trainer.py#L252-L288","documentation":"Raised by BaseTrainer._setup_ddp() when distributed training is launched on multiple Intel XPU (GPU) devices but this PyTorch build lacks the XCCL collective-communication library that multi-XPU DDP requires. The code explicitly checks torch.distributed.is_xccl_available() before selecting the 'xccl' backend for dist.init_process_group(). Without XCCL there is no valid backend for cross-XPU gradient synchronization, so training aborts before the process group forms.","triggerScenarios":"Running a command like `yolo train device=xpu:0,1` (or torchrun with >1 XPU rank) on a PyTorch wheel compiled without oneAPI/XCCL support. The xpu path in _setup_ddp is only reached when world_size > 1 and the device string starts with 'xpu'.","commonSituations":"Using a stock upstream PyTorch wheel (pip install torch) on Intel GPUs instead of the Intel-extended intel-extension-for-pytorch / oneCCL builds; upgrading PyTorch to a version that dropped XCCL; CI runners with XPU hardware but a default torch install.","solutions":["Install a PyTorch build with XCCL support (Intel's XPU-enabled torch wheels, e.g. via intel-extension-for-pytorch and oneCCL bindings), then retry the same command.","Verify availability first in Python: import torch.distributed as dist; print(hasattr(dist, 'is_xccl_available') and dist.is_xccl_available()).","If XCCL cannot be installed, fall back to single-XPU training: `yolo train device=xpu:0`.","If you intended CUDA training, check that the device string was not mistyped as 'xpu' when 'cuda' was meant."],"exampleFix":"# before\nyolo train device=xpu:0,1  # RuntimeError: Multi-XPU training requires XCCL\n\n# after\n# single XPU, no collectives needed\nyolo train device=xpu:0","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\n\ndef can_multi_xpu():\n    return hasattr(dist, \"is_xccl_available\") and dist.is_xccl_available()\n\nif not can_multi_xpu():\n    device = \"xpu:0\"  # fall back to single XPU instead of launching DDP","typeGuard":null,"tryCatchPattern":"try:\n    trainer = Model(...).train(device=\"xpu:0,1\", ...)\nexcept RuntimeError as e:\n    if \"XCCL\" in str(e):\n        log.warning(\"XCCL missing; rerunning on single XPU\")\n        Model(...).train(device=\"xpu:0\", ...)\n    else:\n        raise","preventionTips":["On Intel GPUs, install PyTorch builds from the Intel XPU ecosystem (intel-extension-for-pytorch + oneCCL) before attempting multi-device training.","Gate multi-XPU launch behind a dist.is_xccl_available() check in job scripts.","Pin the tested torch/XCCL versions in the environment file to survive upgrades."],"tags":["distributed","xpu","intel-gpu","ddp","environment"],"backgroundTag":null,"analyzedSha":"0449ea011cfd6c9a0d50a0bf1043aca5190cd476","analyzedAt":"2026-08-15T02:34:13.413Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}