{"record":{"id":"42174b5ccdeaeb83","repo":"sgl-project/sglang","slug":"elastic-ep-world-mlp-sync-dp-size-is-out-of-sync","errorCode":null,"errorMessage":"[Elastic EP] WORLD MLP sync dp_size is out of sync: rank={torch.distributed.get_rank(group)} live_dp_size={live_dp_size} effective_ep_size={effective_ep_size} world_size={world_size} server_args_dp_size={dp_size} local_num_tokens={local_num_tokens} local_forward_mode={local_forward_mode}","messagePattern":"\\[Elastic EP\\] WORLD MLP sync dp_size is out of sync: rank=(.+?) live_dp_size=(.+?) effective_ep_size=(.+?) world_size=(.+?) server_args_dp_size=(.+?) local_num_tokens=(.+?) local_forward_mode=(.+?)","errorType":"panic","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/managers/scheduler_components/dp_attn.py","lineNumber":68,"sourceCode":"def _resolve_elastic_world_dp_size(\n    dp_size: int,\n    *,\n    group: torch.distributed.ProcessGroup,\n    local_num_tokens: int,\n    local_forward_mode: int,\n) -> int:\n    if not world_dp_gather_enabled():\n        return dp_size\n\n    from sglang.srt.elastic_ep.elastic_ep import ElasticEPStateManager\n    from sglang.srt.layers.dp_attention import get_attention_dp_size\n\n    live_dp_size = get_attention_dp_size()\n    effective_ep_size = ElasticEPStateManager.get_effective_ep_size()\n    world_size = torch.distributed.get_world_size(group)\n\n    if live_dp_size != effective_ep_size:\n        raise RuntimeError(\n            \"[Elastic EP] WORLD MLP sync dp_size is out of sync: \"\n            f\"rank={torch.distributed.get_rank(group)} \"\n            f\"live_dp_size={live_dp_size} effective_ep_size={effective_ep_size} \"\n            f\"world_size={world_size} server_args_dp_size={dp_size} \"\n            f\"local_num_tokens={local_num_tokens} \"\n            f\"local_forward_mode={local_forward_mode}\"\n        )\n    if live_dp_size > world_size:\n        raise RuntimeError(\n            \"[Elastic EP] WORLD MLP sync dp_size exceeds WORLD size: \"\n            f\"rank={torch.distributed.get_rank(group)} \"\n            f\"live_dp_size={live_dp_size} world_size={world_size} \"\n            f\"effective_ep_size={effective_ep_size}\"\n        )\n\n    return live_dp_size\n\n","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/managers/scheduler_components/dp_attn.py#L50-L86","documentation":"Elastic expert-parallelism consistency check: during the WORLD MLP sync, the live attention DP size does not match the effective EP size tracked by ElasticEPStateManager, indicating the elastic topology state has diverged across ranks/groups.","triggerScenarios":"Running with elastic EP where get_attention_dp_size() differs from ElasticEPStateManager.get_effective_ep_size() when prepare_mlp_sync_batch_raw runs — e.g. ranks rescaled EP but DP attention state (or vice versa) was not updated consistently.","commonSituations":"Mid-flight elastic EP resize where one component's resize completed and another failed or was skipped; partial rollout of elastic EP changes; mismatched server_args across ranks.","solutions":["Ensure all ranks complete the full elastic EP resize protocol (both attention DP and EP state) before scheduling continues","Check rank logs for an earlier failed resize step that left state inconsistent","Restart the engine and retry the resize as a unit; report if it reproduces with a clean topology"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    batch = prepare_mlp_sync_batch_raw(...)\nexcept RuntimeError as e:\n    if 'WORLD MLP sync dp_size is out of sync' in str(e):\n        restart_engine_and_retry()  # topology state is corrupt\n    raise","preventionTips":["Treat elastic EP resize as atomic across ranks; check all-rank success before continuing","Monitor dp_size/ep_size gauges per rank to catch divergence early"],"tags":["distributed","elastic-ep","dp-attention","topology"],"backgroundTag":"distributed-topology-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}