huggingface/transformers · error · ValueError

FSDP is not compatible with continuous batching but got {dev

Error message

FSDP is not compatible with continuous batching but got {device_mesh = }.

What it means

Raised by DistributedHelper.check_device_mesh_for_cb when the device mesh contains an 'fsdp' dimension with size > 1. Continuous batching explicitly does not support FSDP-sharded weights (paged KV cache + FSDP gather semantics conflict), so this is a hard compatibility check.

Source

Thrown at src/transformers/generation/continuous_batching/distributed.py:96

        # These attributes depend on the DP state
        self.dp_rank = self.global_rank // self.tp_size
        self.dp_size = self.world_size // self.tp_size

        # Accumulator to CPU integer comm
        self._cpu_int_acc = torch.tensor([0, 0], dtype=torch.int64, device="cpu")

    @staticmethod
    def check_device_mesh_for_cb(device_mesh: DeviceMesh | None) -> None:
        """Checks the validity of the device mesh for continuous batching."""
        # No device mesh = no distributed = life is good
        if device_mesh is None:
            return None
        # If there are no named dims, we assume it is a TP mesh  # TODO (remi): this might change after distrib rework
        if device_mesh.mesh_dim_names is None:
            return None
        # FSDP is not compatible with continuous batching, so we raise an error if it is used
        if "fsdp" in device_mesh.mesh_dim_names and device_mesh["fsdp"].size() > 1:
            raise ValueError(f"FSDP is not compatible with continuous batching but got {device_mesh = }.")

    @staticmethod
    def extract_tp_mesh(device_mesh: DeviceMesh | None) -> DeviceMesh | None:
        """Extracts the TP mesh from the device mesh if it exists and is non-trivial."""
        if device_mesh is None:
            return None
        # Case: device mesh with no named dims => assumed TP mesh
        if device_mesh.mesh_dim_names is None:
            return device_mesh if device_mesh.size() > 1 else None
        # Case: device mesh with named dims => extract the TP mesh
        if "tp" in device_mesh.mesh_dim_names and device_mesh["tp"].size() > 1:
            return device_mesh["tp"]
        return None

    def infer_if_tp_driver(self) -> bool:
        return self.tp_local_rank == 0

    def destroy_cpu_comm_group(self) -> None:

View on GitHub (pinned to a597f97485)

Solutions

Drop the FSDP dimension: use a pure TP mesh (mesh_dim_names=('tp',) or unnamed) for continuous batching
If sharding is required, reshard/consolidate weights and run TP-only inference
Remove fully_shard/FSDP wrapping before creating the continuous-batching manager

Example fix

# before
mesh = init_device_mesh('cuda', (2, 2), mesh_dim_names=('fsdp', 'tp'))
cfg = ContinuousBatchingConfig(device_mesh=mesh)

# after
mesh = init_device_mesh('cuda', (4,), mesh_dim_names=('tp',))
cfg = ContinuousBatchingConfig(device_mesh=mesh)

Defensive patterns

Strategy: validation

Validate before calling

def mesh_is_cb_compatible(mesh) -> bool:
    if mesh is None or mesh.mesh_dim_names is None:
        return True
    return not ('fsdp' in mesh.mesh_dim_names and mesh['fsdp'].size() > 1)
assert mesh_is_cb_compatible(device_mesh)

Prevention

Use TP-only meshes for continuous batching
Don't wrap models with fully_shard before CB inference
Check mesh_dim_names before passing any mesh

When it happens

Trigger: Building a mesh like init_device_mesh('cuda', (2, 2), mesh_dim_names=('fsdp', 'tp')) and passing it in ContinuousBatchingConfig while the fsdp dim spans >1 rank.

Common situations: Porting an FSDP training/inference script to continuous batching; using fully_shard-wrapped models then requesting the CB manager; generic mesh utilities that always include an fsdp dim.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/361f68b587f30a65. Report an issue: GitHub.