huggingface/transformers · error · ValueError

Sequence parallelism is enabled but no SP process group is a

Error message

Sequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.

What it means

Error "Sequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1." thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepspeed.py:727

    # The model's forward pass receives shift_labels via **kwargs and passes it to the loss function.
    # Both standard transformer models and Liger-patched models handle shift_labels correctly,
    # so we can directly use the computed loss from the model output.
    # See: https://huggingface.co/docs/accelerate/en/concept_guides/sequence_parallelism
    if "labels" not in inputs and "shift_labels" in inputs:
        # DeepSpeed SP Dataloader removes "labels" but we need it, otherwise, we won't compute the loss.
        inputs["labels"] = inputs["shift_labels"]
    outputs = model(**inputs)
    loss = outputs.loss

    # Prefer DeepSpeed SP groups when using Ulysses; otherwise fall back to torch device mesh.
    if pc.sp_backend == "deepspeed" and pc.sp_size > 1:
        from deepspeed.utils import groups

        sp_group = groups._get_sequence_parallel_group()
    elif accelerator.torch_device_mesh is not None:
        sp_group = accelerator.torch_device_mesh["sp"].get_group()
    else:
        raise ValueError(
            "Sequence parallelism is enabled but no SP process group is available. "
            "Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1."
        )
    # differentiable weighted per-shard-loss aggregation across ranks
    losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
    # special dealing with SFT that has prompt tokens that aren't used in loss computation
    good_tokens = (inputs["shift_labels"] != -100).view(-1).sum()
    good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
    losses_stacked = torch.stack(losses_per_rank)
    good_tokens_stacked = torch.stack(good_tokens_per_rank)
    mask = good_tokens_stacked > 0
    safe_losses = torch.where(mask, losses_stacked, torch.zeros_like(losses_stacked))
    total_loss = (safe_losses * good_tokens_stacked).sum()
    total_good_tokens = good_tokens_stacked.sum()
    loss = total_loss / total_good_tokens.clamp(min=1)

    return (loss, outputs) if return_outputs else loss

View on GitHub (pinned to a597f97485)

Solutions

1. Initialize torch_device_mesh before enabling sequence parallelism.
2. Set sp_backend='deepspeed' with sp_size > 1 in the config.

When it happens

Trigger: Raised when sequence parallelism is configured but no SP process group exists.

Common situations: sp_backend/sp_size set without initializing torch_device_mesh or a matching DeepSpeed SP group.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/8b851fc95f8f6748. Report an issue: GitHub.