{"record":{"id":"adf8d16fcadef11e","repo":"invoke-ai/InvokeAI","slug":"failed-to-gather-tensors-e","errorCode":null,"errorMessage":"Failed to gather tensors: {e}","messagePattern":"Failed to gather tensors: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/pid/_src/utils/context_parallel.py","lineNumber":86,"sourceCode":"        cp_group: The process group for checkpoint parallelism.\n\n    Returns:\n        A tensor that is the concatenation of tensors from all ranks in the cp_group.\n\n    Raises:\n        RuntimeError: If the gather operation fails.\n    \"\"\"\n    # Get the world size (number of processes in the group)\n    world_size = get_world_size(cp_group)\n\n    # Create a list to store tensors from all ranks\n    gathered_tensors = [torch.zeros_like(x) for _ in range(world_size)]\n\n    # Gather tensors from all ranks\n    try:\n        all_gather(gathered_tensors, x, group=cp_group)\n    except RuntimeError as e:\n        raise RuntimeError(f\"Failed to gather tensors: {e}\")\n\n    # Concatenate the gathered tensors along the specified dimension\n    return torch.cat(gathered_tensors, dim=seq_dim)\n\n\ndef cat_outputs_cp_with_grad(x: Tensor, seq_dim: int, cp_group: ProcessGroup) -> Tensor:\n    \"\"\"\n    Concatenate outputs from different ranks in the context parallelism group.\n\n    This function gathers tensors from all ranks in the checkpoint parallelism group\n    and concatenates them along the specified sequence dimension.\n\n    It retains computational graph locally for each rank by replacing the portion of the tensor with original output.\n\n    Args:\n        x: Input tensor to be concatenated.\n        seq_dim: The dimension along which to concatenate the tensors (sequence dimension).\n        cp_group: The process group for checkpoint parallelism.","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/pid/_src/utils/context_parallel.py#L68-L104","documentation":"cat_outputs_cp performs an all_gather of each rank's sequence shard across the context-parallel group, then concatenates them. If the collective raises RuntimeError (mismatched tensor shapes/dtypes, a dead or misconfigured process group, NCCL failure), it is re-raised as RuntimeError(\"Failed to gather tensors: ...\") with the original message appended. It means the context-parallel gather step itself failed, not the model math.","triggerScenarios":"Calling cat_outputs_cp(x, seq_dim, cp_group) when ranks contribute tensors of differing shapes, when the cp_group is invalid/torn down, or when NCCL communication fails (timeout, unmatched collectives).","commonSituations":"Uneven sequence lengths across ranks, a rank crashed earlier leaving others waiting on the collective, running CP code without initializing the distributed process group, or mixing gloo/nccl backends incorrectly.","solutions":["Check the inner exception: fix shape/dtype mismatches so all ranks contribute identical tensor shapes","Verify the process group is initialized and healthy on all ranks (dist.is_initialized, matching world size)","Ensure all ranks execute the same number/order of collectives (no early return on one rank)","Pad or re-chunk the sequence so every rank gets an equal shard"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"assert torch.distributed.is_initialized()\nassert x.shape[seq_dim] % cp_group.size() == 0","typeGuard":null,"tryCatchPattern":"try:\n    out = cat_outputs_cp(x, seq_dim, cp_group)\nexcept RuntimeError as e:\n    if \"Failed to gather tensors\" in str(e):\n        logger.error(f\"CP gather failed: {e}\")\n    raise","preventionTips":["Equalize shard shapes across ranks","Use torchrun so rank failures abort cleanly","Log per-rank shapes before collectives","Smoke-test with world_size=2"],"tags":["context-parallel","all-gather","distributed","nccl","runtime-error"],"backgroundTag":"distributed-collective-failure","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}