invoke-ai/InvokeAI · error · RuntimeError

Failed to gather tensors: {e}

Error message

Failed to gather tensors: {e}

What it means

cat_outputs_cp performs an all_gather of each rank's sequence shard across the context-parallel group, then concatenates them. If the collective raises RuntimeError (mismatched tensor shapes/dtypes, a dead or misconfigured process group, NCCL failure), it is re-raised as RuntimeError("Failed to gather tensors: ...") with the original message appended. It means the context-parallel gather step itself failed, not the model math.

Source

Thrown at invokeai/backend/pid/_src/utils/context_parallel.py:86

        cp_group: The process group for checkpoint parallelism.

    Returns:
        A tensor that is the concatenation of tensors from all ranks in the cp_group.

    Raises:
        RuntimeError: If the gather operation fails.
    """
    # Get the world size (number of processes in the group)
    world_size = get_world_size(cp_group)

    # Create a list to store tensors from all ranks
    gathered_tensors = [torch.zeros_like(x) for _ in range(world_size)]

    # Gather tensors from all ranks
    try:
        all_gather(gathered_tensors, x, group=cp_group)
    except RuntimeError as e:
        raise RuntimeError(f"Failed to gather tensors: {e}")

    # Concatenate the gathered tensors along the specified dimension
    return torch.cat(gathered_tensors, dim=seq_dim)


def cat_outputs_cp_with_grad(x: Tensor, seq_dim: int, cp_group: ProcessGroup) -> Tensor:
    """
    Concatenate outputs from different ranks in the context parallelism group.

    This function gathers tensors from all ranks in the checkpoint parallelism group
    and concatenates them along the specified sequence dimension.

    It retains computational graph locally for each rank by replacing the portion of the tensor with original output.

    Args:
        x: Input tensor to be concatenated.
        seq_dim: The dimension along which to concatenate the tensors (sequence dimension).
        cp_group: The process group for checkpoint parallelism.

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Check the inner exception: fix shape/dtype mismatches so all ranks contribute identical tensor shapes
2. Verify the process group is initialized and healthy on all ranks (dist.is_initialized, matching world size)
3. Ensure all ranks execute the same number/order of collectives (no early return on one rank)
4. Pad or re-chunk the sequence so every rank gets an equal shard

assert torch.distributed.is_initialized()
assert x.shape[seq_dim] % cp_group.size() == 0

try:
    out = cat_outputs_cp(x, seq_dim, cp_group)
except RuntimeError as e:
    if "Failed to gather tensors" in str(e):
        logger.error(f"CP gather failed: {e}")
    raise

When it happens

Trigger: Calling cat_outputs_cp(x, seq_dim, cp_group) when ranks contribute tensors of differing shapes, when the cp_group is invalid/torn down, or when NCCL communication fails (timeout, unmatched collectives).

Common situations: Uneven sequence lengths across ranks, a rank crashed earlier leaving others waiting on the collective, running CP code without initializing the distributed process group, or mixing gloo/nccl backends incorrectly.

Related errors

• Mistral encoder did not return hidden_states. Ensure output_
• Qwen3-VL encoder did not return hidden_states; cannot build
• Text encoder did not return hidden_states. Ensure output_hid
• Expected at least 2 hidden states from text encoder, got {le
• JWT secret not found in database. This should have been crea

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/adf8d16fcadef11e. Report an issue: GitHub.