{"record":{"id":"8b851fc95f8f6748","repo":"huggingface/transformers","slug":"sequence-parallelism-is-enabled-but-no-sp-process","errorCode":null,"errorMessage":"Sequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.","messagePattern":"Sequence parallelism is enabled but no SP process group is available\\. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepspeed.py","lineNumber":727,"sourceCode":"    # The model's forward pass receives shift_labels via **kwargs and passes it to the loss function.\n    # Both standard transformer models and Liger-patched models handle shift_labels correctly,\n    # so we can directly use the computed loss from the model output.\n    # See: https://huggingface.co/docs/accelerate/en/concept_guides/sequence_parallelism\n    if \"labels\" not in inputs and \"shift_labels\" in inputs:\n        # DeepSpeed SP Dataloader removes \"labels\" but we need it, otherwise, we won't compute the loss.\n        inputs[\"labels\"] = inputs[\"shift_labels\"]\n    outputs = model(**inputs)\n    loss = outputs.loss\n\n    # Prefer DeepSpeed SP groups when using Ulysses; otherwise fall back to torch device mesh.\n    if pc.sp_backend == \"deepspeed\" and pc.sp_size > 1:\n        from deepspeed.utils import groups\n\n        sp_group = groups._get_sequence_parallel_group()\n    elif accelerator.torch_device_mesh is not None:\n        sp_group = accelerator.torch_device_mesh[\"sp\"].get_group()\n    else:\n        raise ValueError(\n            \"Sequence parallelism is enabled but no SP process group is available. \"\n            \"Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.\"\n        )\n    # differentiable weighted per-shard-loss aggregation across ranks\n    losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)\n    # special dealing with SFT that has prompt tokens that aren't used in loss computation\n    good_tokens = (inputs[\"shift_labels\"] != -100).view(-1).sum()\n    good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)\n    losses_stacked = torch.stack(losses_per_rank)\n    good_tokens_stacked = torch.stack(good_tokens_per_rank)\n    mask = good_tokens_stacked > 0\n    safe_losses = torch.where(mask, losses_stacked, torch.zeros_like(losses_stacked))\n    total_loss = (safe_losses * good_tokens_stacked).sum()\n    total_good_tokens = good_tokens_stacked.sum()\n    loss = total_loss / total_good_tokens.clamp(min=1)\n\n    return (loss, outputs) if return_outputs else loss\n","sourceCodeStart":709,"sourceCodeEnd":745,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepspeed.py#L709-L745","documentation":"Error \"Sequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.\" thrown in huggingface/transformers.","triggerScenarios":"Raised when sequence parallelism is configured but no SP process group exists.","commonSituations":"sp_backend/sp_size set without initializing torch_device_mesh or a matching DeepSpeed SP group.","solutions":["Initialize torch_device_mesh before enabling sequence parallelism.","Set sp_backend='deepspeed' with sp_size > 1 in the config."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}