{"record":{"id":"290c572c51aa73ca","repo":"vllm-project/vllm","slug":"self-communicator-communicator-is-incompatible-w","errorCode":null,"errorMessage":"{self.communicator} communicator is incompatible with async EPLB due to NCCL multi-stream conflicts. Use 'torch_gloo' or 'nixl' instead, or leave communicator unset for automatic selection.","messagePattern":"(.+?) communicator is incompatible with async EPLB due to NCCL multi-stream conflicts\\. Use 'torch_gloo' or 'nixl' instead, or leave communicator unset for automatic selection\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/parallel.py","lineNumber":107,"sourceCode":"    policy: EPLBPolicyOption = \"default\"\n    \"\"\"The policy type for expert parallel load balancing (EPLB).\"\"\"\n\n    communicator: EPLBCommunicatorBackend | None = None\n    \"\"\"\n    Backend for EPLB expert weight communication:\n    - \"torch_nccl\": Use torch.distributed on the device process group\n    - \"torch_gloo\": Use torch.distributed gloo with CPU staging\n    - \"nixl\": Use NIXL with staged send/recv buffers\n    - \"pynccl\": Use PyNccl send/recv\n    - None: Auto-select backend (prefers \"nixl\", falls back to \"torch_gloo\")\n    \"\"\"\n\n    @model_validator(mode=\"after\")\n    def _validate_eplb_config(self) -> Self:\n        if self.use_async and self.policy != \"default\":\n            raise ValueError(\"Async EPLB is only supported with the default policy.\")\n        if self.use_async and self.communicator in (\"torch_nccl\", \"pynccl\"):\n            raise ValueError(\n                f\"{self.communicator} communicator is incompatible with \"\n                \"async EPLB due to NCCL multi-stream conflicts. Use \"\n                \"'torch_gloo' or 'nixl' instead, or leave communicator \"\n                \"unset for automatic selection.\"\n            )\n        if self.log_balancedness and self.log_balancedness_interval <= 0:\n            raise ValueError(\"log_balancedness_interval must be greater than 0.\")\n        return self\n\n\n@config\nclass ParallelConfig:\n    \"\"\"Configuration for the distributed execution.\"\"\"\n\n    pipeline_parallel_size: int = Field(default=1, ge=1)\n    \"\"\"Number of pipeline parallel groups.\"\"\"\n    tensor_parallel_size: int = Field(default=1, ge=1)\n    \"\"\"Number of tensor parallel groups.\"\"\"","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/parallel.py#L89-L125","documentation":"EPLBConfig's validator forbids use_async=True with communicator 'torch_nccl' or 'pynccl'. Both communicators issue NCCL operations on the device stream, and async EPLB moves expert weights while compute streams are active, causing NCCL multi-stream conflicts (deadlocks/corruption). Use 'torch_gloo' or 'nixl', or leave the communicator unset for auto-selection (which prefers nixl, falling back to torch_gloo).","triggerScenarios":"Passing --enable-eplb --async-eplb --eplb-communicator torch_nccl (or pynccl); or a config file where communicator was pinned for the sync path and --async-eplb was added later.","commonSituations":"Reusing a working sync-EPLB config when enabling async EPLB; explicitly choosing torch_nccl believing it is the highest-performance option.","solutions":["Remove --eplb-communicator so vLLM auto-selects (nixl preferred, torch_gloo fallback).","Or set --eplb-communicator torch_gloo or --eplb-communicator nixl explicitly.","If you truly need torch_nccl/pynccl, disable --async-eplb."],"exampleFix":"# before\nvllm serve DeepSeek-ai/DeepSeek-V3 --enable-eplb --async-eplb --eplb-communicator torch_nccl\n\n# after\nvllm serve DeepSeek-ai/DeepSeek-V3 --enable-eplb --async-eplb --eplb-communicator nixl","handlingStrategy":"validation","validationCode":"def check_eplb_communicator(use_async: bool, communicator: str | None) -> None:\n    if use_async and communicator in (\"torch_nccl\", \"pynccl\"):\n        raise SystemExit(\"Use torch_gloo/nixl (or unset) with async EPLB\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Leave --eplb-communicator unset to get safe auto-selection.","Regression-test the full EPLB flag set after upgrading vLLM; communicator constraints change."],"tags":["vllm","config","eplb","nccl","moe","validation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}