{"record":{"id":"de3ff2a40cfe9a12","repo":"xai-org/x-algorithm","slug":"async-emb-world-size-changed-from-world-size-t","errorCode":null,"errorMessage":"async_emb world size changed from ${world_size_} to ${world_size}","messagePattern":"async_emb world size changed from (.+?) to (.+?)","errorType":"exception","errorClass":"std::invalid_argument","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":234,"sourceCode":"    LOG(ERROR) << \"ncclCommFinalize failed: \" << ncclGetErrorString(result);\n    abortCommunicator(\"ncclCommFinalize failed\");\n    comm_ = nullptr;\n    return;\n  }\n\n  result = ncclCommDestroy(comm_);\n  if (result != ncclSuccess) {\n    LOG(ERROR) << \"ncclCommDestroy failed: \" << ncclGetErrorString(result);\n  }\n  comm_ = nullptr;\n}\n\nstd::vector<std::vector<uint8_t>> AsyncEmbContext::reset(int rank, int world_size) {\n  if (world_size <= 0 || rank < 0 || rank >= world_size) {\n    throw std::invalid_argument(\"async_emb received an invalid rank or world size\");\n  }\n  if (world_size != world_size_) {\n    throw std::invalid_argument(\n        \"async_emb world size changed from \" + std::to_string(world_size_) + \" to \" +\n        std::to_string(world_size)\n    );\n  }\n  if ((world_size & (world_size - 1)) != 0) {\n    throw std::invalid_argument(\"async_emb currently requires power-of-two EP\");\n  }\n\n  rank_ = rank;\n  initialized_ = false;\n  std::vector<std::vector<uint8_t>> bootstrap(world_size);\n  if (rank != 0) {\n    return bootstrap;\n  }\n\n  ncclUniqueId id;\n  ncclResult_t result = ncclGetUniqueId(&id);\n  if (result != ncclSuccess) {","sourceCodeStart":216,"sourceCodeEnd":252,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L216-L252","documentation":"The context fixes the expert-parallel world size at construction; reset() rejects a different value because the arena layout and NCCL communicator were sized for the original world_size_. This prevents subtle corruption from re-initializing with a mismatched topology.","triggerScenarios":"Calling reset(rank, n) with a different n than the AsyncEmbContext was constructed with — typically after a elastic restart where some ranks rejoined with a new world size, or when two different jobs share a context object.","commonSituations":"TorchElastic / torchrun restart changes world size between attempts; unit tests reusing a global context across test cases with different worker counts.","solutions":["Destroy and reconstruct AsyncEmbContext when the world size genuinely changes.","Ensure every rank passes the same world_size derived from one process-group initialization.","In elastic setups, allocate a fresh context per resize event rather than calling reset with new topology."],"exampleFix":"// before\nctx.reset(rank, new_world_size);  // throws\n\n// after\nctx = std::make_unique<AsyncEmbContext>(new_world_size);\nctx->reset(rank, new_world_size);","handlingStrategy":"fallback","validationCode":"if (new_world_size != constructed_world_size) {\n    // rebuild instead of reset\n    ctx = std::make_unique<AsyncEmbContext>(new_world_size);\n}","typeGuard":null,"tryCatchPattern":"catch (const std::invalid_argument& e) { if rebuilt_needed -> recreate context; }","preventionTips":["Treat world size as immutable per context; document it at construction.","In elastic jobs, recreate contexts on resize events."],"tags":["world-size","distributed","lifecycle","elastic","async-emb"],"backgroundTag":"state-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}