huggingface/transformers · error · ValueError

Distributed is off but received {device_mesh = }.

Error message

Distributed is off but received {device_mesh = }.

What it means

Raised by DistributedHelper.__init__ when a non-trivial TP mesh was extracted from the provided device_mesh but torch.distributed is not initialized. Continuous batching treats a device mesh as a distributed-instruction; handing one to a single-process run is a configuration contradiction.

Source

Thrown at src/transformers/generation/continuous_batching/distributed.py:51

    DeviceMesh = object  # only used for type checking, so this is ok


T = TypeVar("T")


class DistributedHelper:
    """A helper class to handle distributed-related operations. Notably, it does not crash when distributed is off."""

    def __init__(self, device_mesh: DeviceMesh | None, cpu_group_timeout: float | None) -> None:
        self.dist_on = _is_torch_distributed_initialized()
        self.device_mesh = device_mesh

        # Check validity of the device mesh
        self.check_device_mesh_for_cb(self.device_mesh)
        # Extract a non-trivial TP mesh if it exists
        tp_mesh = self.extract_tp_mesh(self.device_mesh)
        if tp_mesh is not None and not self.dist_on:
            raise ValueError(f"Distributed is off but received {device_mesh = }.")

        # These attributes depend on the global dist state
        self.global_rank = dist.get_rank() if self.dist_on else 0
        self.world_size = dist.get_world_size() if self.dist_on else 1

        # These attributes depend on the TP state
        if tp_mesh is not None:
            self.tp_size = tp_mesh.size()
            self.tp_group = tp_mesh.get_group()
            self.tp_root_global_rank = dist.get_global_rank(self.tp_group, 0)
            self.tp_local_rank = tp_mesh.get_local_rank()
            # If TP is on, we create a dedicated CPU group, with an eventual timeout
            tp_ranks = dist.get_process_group_ranks(self.tp_group)
            timeout = None if cpu_group_timeout is None else timedelta(seconds=cpu_group_timeout)
            self.cpu_comm_group = dist.new_group(ranks=tp_ranks, backend="gloo", timeout=timeout)
        else:
            self.tp_size = 1
            self.tp_group = None

View on GitHub (pinned to a597f97485)

Solutions

1. Do not pass device_mesh when running single-process / distributed off
2. Initialize the process group before creating/passing the mesh: torch.distributed.init_process_group(backend='nccl')
3. Gate the mesh: device_mesh = mesh if torch.distributed.is_initialized() else None

Example fix

# before
cfg = ContinuousBatchingConfig(device_mesh=my_mesh)  # dist not initialized

# after
cfg = ContinuousBatchingConfig(device_mesh=my_mesh if torch.distributed.is_initialized() else None)

import torch.distributed as dist
if device_mesh is not None and not dist.is_available() or not dist.is_initialized():
    device_mesh = None  # single-process run: drop the mesh

When it happens

Trigger: Passing continuous_batching_config with device_mesh=init_device_mesh('cuda', (2,)) while torch.distributed.init_process_group() was never called (or dist.is_initialized() is False on this rank).

Common situations: Reusing a config object created for a multi-GPU script in a single-GPU test; device mesh created before process-group init; torchrun vs plain python launch mismatch.

Related errors

• FSDP+TP is not supported yet. Use DistributedConfig(fsdp_siz
• save_pretrained(..., distributed_checkpoint=True) requires t
• m must be provided if max_batch_tokens and num_blocks are No
• Memory footprint {} is more than available memory {}
• Invalid values: max_batch_tokens = {}, num_blocks = {}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/2afc8d99cff48b59. Report an issue: GitHub.