{"record":{"id":"2afc8d99cff48b59","repo":"huggingface/transformers","slug":"distributed-is-off-but-received-device-mesh","errorCode":null,"errorMessage":"Distributed is off but received {device_mesh = }.","messagePattern":"Distributed is off but received (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/distributed.py","lineNumber":51,"sourceCode":"    DeviceMesh = object  # only used for type checking, so this is ok\n\n\nT = TypeVar(\"T\")\n\n\nclass DistributedHelper:\n    \"\"\"A helper class to handle distributed-related operations. Notably, it does not crash when distributed is off.\"\"\"\n\n    def __init__(self, device_mesh: DeviceMesh | None, cpu_group_timeout: float | None) -> None:\n        self.dist_on = _is_torch_distributed_initialized()\n        self.device_mesh = device_mesh\n\n        # Check validity of the device mesh\n        self.check_device_mesh_for_cb(self.device_mesh)\n        # Extract a non-trivial TP mesh if it exists\n        tp_mesh = self.extract_tp_mesh(self.device_mesh)\n        if tp_mesh is not None and not self.dist_on:\n            raise ValueError(f\"Distributed is off but received {device_mesh = }.\")\n\n        # These attributes depend on the global dist state\n        self.global_rank = dist.get_rank() if self.dist_on else 0\n        self.world_size = dist.get_world_size() if self.dist_on else 1\n\n        # These attributes depend on the TP state\n        if tp_mesh is not None:\n            self.tp_size = tp_mesh.size()\n            self.tp_group = tp_mesh.get_group()\n            self.tp_root_global_rank = dist.get_global_rank(self.tp_group, 0)\n            self.tp_local_rank = tp_mesh.get_local_rank()\n            # If TP is on, we create a dedicated CPU group, with an eventual timeout\n            tp_ranks = dist.get_process_group_ranks(self.tp_group)\n            timeout = None if cpu_group_timeout is None else timedelta(seconds=cpu_group_timeout)\n            self.cpu_comm_group = dist.new_group(ranks=tp_ranks, backend=\"gloo\", timeout=timeout)\n        else:\n            self.tp_size = 1\n            self.tp_group = None","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/distributed.py#L33-L69","documentation":"Raised by DistributedHelper.__init__ when a non-trivial TP mesh was extracted from the provided device_mesh but torch.distributed is not initialized. Continuous batching treats a device mesh as a distributed-instruction; handing one to a single-process run is a configuration contradiction.","triggerScenarios":"Passing continuous_batching_config with device_mesh=init_device_mesh('cuda', (2,)) while torch.distributed.init_process_group() was never called (or dist.is_initialized() is False on this rank).","commonSituations":"Reusing a config object created for a multi-GPU script in a single-GPU test; device mesh created before process-group init; torchrun vs plain python launch mismatch.","solutions":["Do not pass device_mesh when running single-process / distributed off","Initialize the process group before creating/passing the mesh: torch.distributed.init_process_group(backend='nccl')","Gate the mesh: device_mesh = mesh if torch.distributed.is_initialized() else None"],"exampleFix":"# before\ncfg = ContinuousBatchingConfig(device_mesh=my_mesh)  # dist not initialized\n\n# after\ncfg = ContinuousBatchingConfig(device_mesh=my_mesh if torch.distributed.is_initialized() else None)","handlingStrategy":"validation","validationCode":"import torch.distributed as dist\nif device_mesh is not None and not dist.is_available() or not dist.is_initialized():\n    device_mesh = None  # single-process run: drop the mesh","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only build device meshes after init_process_group","Gate mesh creation on dist.is_initialized()","Don't share config objects between single- and multi-process scripts"],"tags":["distributed","device-mesh","configuration","continuous-batching"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}