{"record":{"id":"8140d70640298d52","repo":"xai-org/x-algorithm","slug":"async-emb-currently-requires-power-of-two-ep","errorCode":null,"errorMessage":"async_emb currently requires power-of-two EP","messagePattern":"async_emb currently requires power-of-two EP","errorType":"exception","errorClass":"std::invalid_argument","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":240,"sourceCode":"  result = ncclCommDestroy(comm_);\n  if (result != ncclSuccess) {\n    LOG(ERROR) << \"ncclCommDestroy failed: \" << ncclGetErrorString(result);\n  }\n  comm_ = nullptr;\n}\n\nstd::vector<std::vector<uint8_t>> AsyncEmbContext::reset(int rank, int world_size) {\n  if (world_size <= 0 || rank < 0 || rank >= world_size) {\n    throw std::invalid_argument(\"async_emb received an invalid rank or world size\");\n  }\n  if (world_size != world_size_) {\n    throw std::invalid_argument(\n        \"async_emb world size changed from \" + std::to_string(world_size_) + \" to \" +\n        std::to_string(world_size)\n    );\n  }\n  if ((world_size & (world_size - 1)) != 0) {\n    throw std::invalid_argument(\"async_emb currently requires power-of-two EP\");\n  }\n\n  rank_ = rank;\n  initialized_ = false;\n  std::vector<std::vector<uint8_t>> bootstrap(world_size);\n  if (rank != 0) {\n    return bootstrap;\n  }\n\n  ncclUniqueId id;\n  ncclResult_t result = ncclGetUniqueId(&id);\n  if (result != ncclSuccess) {\n    throw ncclError(\"ncclGetUniqueId\", result);\n  }\n\n  std::vector<uint8_t> bytes(sizeof(id));\n  std::memcpy(bytes.data(), &id, sizeof(id));\n  std::fill(bootstrap.begin(), bootstrap.end(), bytes);","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L222-L258","documentation":"The internal all-to-all scheduling assumes a power-of-two expert-parallel degree, so reset() checks (world_size & (world_size-1)) == 0 and rejects non power-of-two values such as 3, 5, 6, 12. This is a documented algorithmic limitation, not a transient fault.","triggerScenarios":"Calling reset with world_size values like 3, 5, 6, 7, 9, 12... — e.g. requesting 6 GPUs for EP instead of 4 or 8.","commonSituations":"Cluster allocations giving odd GPU counts; scaling experiments (12-way EP) that work in other libraries but not here; mixing data-parallel and expert-parallel ranks so the EP subgroup size is non power-of-two.","solutions":["Set expert-parallel world size to the nearest power of two (1, 2, 4, 8, 16, 32).","Restructure the job so the EP group is a dedicated power-of-two subgroup and other ranks are data-parallel.","Request node shapes that yield power-of-two total GPUs (e.g. 8-GPU nodes)."],"exampleFix":"# before\ntorchrun --nproc_per_node=6 train.py  # EP=6 -> throws\n\n# after\ntorchrun --nproc_per_node=8 train.py  # EP=8 -> ok","handlingStrategy":"validation","validationCode":"bool pow2(int n) { return n > 0 && (n & (n - 1)) == 0; }\nif (!pow2(ep_world_size))\n    throw std::invalid_argument(\"EP world size must be a power of two\");","typeGuard":"bool isPow2(int n);","tryCatchPattern":null,"preventionTips":["Provision power-of-two GPU counts for expert-parallel groups.","Fail fast in job configs with a power-of-two assertion."],"tags":["world-size","distributed","constraint","expert-parallel","async-emb"],"backgroundTag":"unsupported-configuration","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}