sgl-project/sglang · critical · RuntimeError

[Elastic EP] WORLD MLP sync dp_size is out of sync: rank={to

Error message

[Elastic EP] WORLD MLP sync dp_size is out of sync: rank={torch.distributed.get_rank(group)} live_dp_size={live_dp_size} effective_ep_size={effective_ep_size} world_size={world_size} server_args_dp_size={dp_size} local_num_tokens={local_num_tokens} local_forward_mode={local_forward_mode}

What it means

Elastic expert-parallelism consistency check: during the WORLD MLP sync, the live attention DP size does not match the effective EP size tracked by ElasticEPStateManager, indicating the elastic topology state has diverged across ranks/groups.

Source

Thrown at python/sglang/srt/managers/scheduler_components/dp_attn.py:68

def _resolve_elastic_world_dp_size(
    dp_size: int,
    *,
    group: torch.distributed.ProcessGroup,
    local_num_tokens: int,
    local_forward_mode: int,
) -> int:
    if not world_dp_gather_enabled():
        return dp_size

    from sglang.srt.elastic_ep.elastic_ep import ElasticEPStateManager
    from sglang.srt.layers.dp_attention import get_attention_dp_size

    live_dp_size = get_attention_dp_size()
    effective_ep_size = ElasticEPStateManager.get_effective_ep_size()
    world_size = torch.distributed.get_world_size(group)

    if live_dp_size != effective_ep_size:
        raise RuntimeError(
            "[Elastic EP] WORLD MLP sync dp_size is out of sync: "
            f"rank={torch.distributed.get_rank(group)} "
            f"live_dp_size={live_dp_size} effective_ep_size={effective_ep_size} "
            f"world_size={world_size} server_args_dp_size={dp_size} "
            f"local_num_tokens={local_num_tokens} "
            f"local_forward_mode={local_forward_mode}"
        )
    if live_dp_size > world_size:
        raise RuntimeError(
            "[Elastic EP] WORLD MLP sync dp_size exceeds WORLD size: "
            f"rank={torch.distributed.get_rank(group)} "
            f"live_dp_size={live_dp_size} world_size={world_size} "
            f"effective_ep_size={effective_ep_size}"
        )

    return live_dp_size

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure all ranks complete the full elastic EP resize protocol (both attention DP and EP state) before scheduling continues
2. Check rank logs for an earlier failed resize step that left state inconsistent
3. Restart the engine and retry the resize as a unit; report if it reproduces with a clean topology

try:
    batch = prepare_mlp_sync_batch_raw(...)
except RuntimeError as e:
    if 'WORLD MLP sync dp_size is out of sync' in str(e):
        restart_engine_and_retry()  # topology state is corrupt
    raise

When it happens

Trigger: Running with elastic EP where get_attention_dp_size() differs from ElasticEPStateManager.get_effective_ep_size() when prepare_mlp_sync_batch_raw runs — e.g. ranks rescaled EP but DP attention state (or vice versa) was not updated consistently.

Common situations: Mid-flight elastic EP resize where one component's resize completed and another failed or was skipped; partial rollout of elastic EP changes; mismatched server_args across ranks.

Related errors

• [Elastic EP] WORLD MLP sync dp_size exceeds WORLD size: rank
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.
• world_size ({world_size}) is less than tensor_parallel_degre
• {type(self).__name__} does not implement ring KV-chunk atten

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/42174b5ccdeaeb83. Report an issue: GitHub.