{"record":{"id":"361f68b587f30a65","repo":"huggingface/transformers","slug":"fsdp-is-not-compatible-with-continuous-batching-bu","errorCode":null,"errorMessage":"FSDP is not compatible with continuous batching but got {device_mesh = }.","messagePattern":"FSDP is not compatible with continuous batching but got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/continuous_batching/distributed.py","lineNumber":96,"sourceCode":"        # These attributes depend on the DP state\n        self.dp_rank = self.global_rank // self.tp_size\n        self.dp_size = self.world_size // self.tp_size\n\n        # Accumulator to CPU integer comm\n        self._cpu_int_acc = torch.tensor([0, 0], dtype=torch.int64, device=\"cpu\")\n\n    @staticmethod\n    def check_device_mesh_for_cb(device_mesh: DeviceMesh | None) -> None:\n        \"\"\"Checks the validity of the device mesh for continuous batching.\"\"\"\n        # No device mesh = no distributed = life is good\n        if device_mesh is None:\n            return None\n        # If there are no named dims, we assume it is a TP mesh  # TODO (remi): this might change after distrib rework\n        if device_mesh.mesh_dim_names is None:\n            return None\n        # FSDP is not compatible with continuous batching, so we raise an error if it is used\n        if \"fsdp\" in device_mesh.mesh_dim_names and device_mesh[\"fsdp\"].size() > 1:\n            raise ValueError(f\"FSDP is not compatible with continuous batching but got {device_mesh = }.\")\n\n    @staticmethod\n    def extract_tp_mesh(device_mesh: DeviceMesh | None) -> DeviceMesh | None:\n        \"\"\"Extracts the TP mesh from the device mesh if it exists and is non-trivial.\"\"\"\n        if device_mesh is None:\n            return None\n        # Case: device mesh with no named dims => assumed TP mesh\n        if device_mesh.mesh_dim_names is None:\n            return device_mesh if device_mesh.size() > 1 else None\n        # Case: device mesh with named dims => extract the TP mesh\n        if \"tp\" in device_mesh.mesh_dim_names and device_mesh[\"tp\"].size() > 1:\n            return device_mesh[\"tp\"]\n        return None\n\n    def infer_if_tp_driver(self) -> bool:\n        return self.tp_local_rank == 0\n\n    def destroy_cpu_comm_group(self) -> None:","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/continuous_batching/distributed.py#L78-L114","documentation":"Raised by DistributedHelper.check_device_mesh_for_cb when the device mesh contains an 'fsdp' dimension with size > 1. Continuous batching explicitly does not support FSDP-sharded weights (paged KV cache + FSDP gather semantics conflict), so this is a hard compatibility check.","triggerScenarios":"Building a mesh like init_device_mesh('cuda', (2, 2), mesh_dim_names=('fsdp', 'tp')) and passing it in ContinuousBatchingConfig while the fsdp dim spans >1 rank.","commonSituations":"Porting an FSDP training/inference script to continuous batching; using fully_shard-wrapped models then requesting the CB manager; generic mesh utilities that always include an fsdp dim.","solutions":["Drop the FSDP dimension: use a pure TP mesh (mesh_dim_names=('tp',) or unnamed) for continuous batching","If sharding is required, reshard/consolidate weights and run TP-only inference","Remove fully_shard/FSDP wrapping before creating the continuous-batching manager"],"exampleFix":"# before\nmesh = init_device_mesh('cuda', (2, 2), mesh_dim_names=('fsdp', 'tp'))\ncfg = ContinuousBatchingConfig(device_mesh=mesh)\n\n# after\nmesh = init_device_mesh('cuda', (4,), mesh_dim_names=('tp',))\ncfg = ContinuousBatchingConfig(device_mesh=mesh)","handlingStrategy":"validation","validationCode":"def mesh_is_cb_compatible(mesh) -> bool:\n    if mesh is None or mesh.mesh_dim_names is None:\n        return True\n    return not ('fsdp' in mesh.mesh_dim_names and mesh['fsdp'].size() > 1)\nassert mesh_is_cb_compatible(device_mesh)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use TP-only meshes for continuous batching","Don't wrap models with fully_shard before CB inference","Check mesh_dim_names before passing any mesh"],"tags":["distributed","fsdp","compatibility","continuous-batching"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}