{"record":{"id":"ebb00cc29f46a555","repo":"sgl-project/sglang","slug":"elastic-ep-world-mlp-sync-dp-size-exceeds-world","errorCode":null,"errorMessage":"[Elastic EP] WORLD MLP sync dp_size exceeds WORLD size: rank={torch.distributed.get_rank(group)} live_dp_size={live_dp_size} world_size={world_size} effective_ep_size={effective_ep_size}","messagePattern":"\\[Elastic EP\\] WORLD MLP sync dp_size exceeds WORLD size: rank=(.+?) live_dp_size=(.+?) world_size=(.+?) effective_ep_size=(.+?)","errorType":"panic","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/managers/scheduler_components/dp_attn.py","lineNumber":77,"sourceCode":"\n    from sglang.srt.elastic_ep.elastic_ep import ElasticEPStateManager\n    from sglang.srt.layers.dp_attention import get_attention_dp_size\n\n    live_dp_size = get_attention_dp_size()\n    effective_ep_size = ElasticEPStateManager.get_effective_ep_size()\n    world_size = torch.distributed.get_world_size(group)\n\n    if live_dp_size != effective_ep_size:\n        raise RuntimeError(\n            \"[Elastic EP] WORLD MLP sync dp_size is out of sync: \"\n            f\"rank={torch.distributed.get_rank(group)} \"\n            f\"live_dp_size={live_dp_size} effective_ep_size={effective_ep_size} \"\n            f\"world_size={world_size} server_args_dp_size={dp_size} \"\n            f\"local_num_tokens={local_num_tokens} \"\n            f\"local_forward_mode={local_forward_mode}\"\n        )\n    if live_dp_size > world_size:\n        raise RuntimeError(\n            \"[Elastic EP] WORLD MLP sync dp_size exceeds WORLD size: \"\n            f\"rank={torch.distributed.get_rank(group)} \"\n            f\"live_dp_size={live_dp_size} world_size={world_size} \"\n            f\"effective_ep_size={effective_ep_size}\"\n        )\n\n    return live_dp_size\n\n\n@dataclass\nclass MLPSyncBatchInfo:\n    dp_size: int\n    tp_size: int\n    cp_size: int\n\n    num_tokens: int\n    num_tokens_for_logprob: int\n    can_run_decode_cuda_graph: bool","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/managers/scheduler_components/dp_attn.py#L59-L95","documentation":"Elastic EP guard: the live attention DP size exceeds the WORLD (process group) size, which is topologically impossible and means the elastic state over-counts replicas relative to actual ranks.","triggerScenarios":"get_attention_dp_size() > torch.distributed.get_world_size(group) during prepare_mlp_sync_batch_raw, e.g. after a buggy resize that set dp_size beyond the launched world, or launching with dp_size larger than the number of GPUs/ranks.","commonSituations":"Misconfigured --dp-size vs --tensor-parallel-size/--expert-parallel-size world size; elastic resize arithmetic bugs; env var overrides applied on only some ranks.","solutions":["Verify dp_size * tp/ep configuration divides the actual world size (GPU count) exactly","Recheck elastic EP resize parameters and that no rank applied a stale dp_size","Relaunch with a consistent topology"],"exampleFix":"# before\n--dp-size 8 --tp 8  # 8 dp ranks needed but world has 8 total\n# after\n--dp-size 4 --tp 8  # or add GPUs so dp_size <= world_size","handlingStrategy":"validation","validationCode":"world = torch.distributed.get_world_size()\nassert dp_size <= world, f'dp_size {dp_size} exceeds world {world}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate dp_size * tp_size divides total GPU count at launch","Use a topology-check script before starting multi-node jobs"],"tags":["distributed","elastic-ep","dp-size","topology"],"backgroundTag":"distributed-topology-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}