xai-org/x-algorithm · error · std::invalid_argument

async_emb world size changed from ${world_size_} to ${world_

Error message

async_emb world size changed from ${world_size_} to ${world_size}

What it means

The context fixes the expert-parallel world size at construction; reset() rejects a different value because the arena layout and NCCL communicator were sized for the original world_size_. This prevents subtle corruption from re-initializing with a mismatched topology.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:234

    LOG(ERROR) << "ncclCommFinalize failed: " << ncclGetErrorString(result);
    abortCommunicator("ncclCommFinalize failed");
    comm_ = nullptr;
    return;
  }

  result = ncclCommDestroy(comm_);
  if (result != ncclSuccess) {
    LOG(ERROR) << "ncclCommDestroy failed: " << ncclGetErrorString(result);
  }
  comm_ = nullptr;
}

std::vector<std::vector<uint8_t>> AsyncEmbContext::reset(int rank, int world_size) {
  if (world_size <= 0 || rank < 0 || rank >= world_size) {
    throw std::invalid_argument("async_emb received an invalid rank or world size");
  }
  if (world_size != world_size_) {
    throw std::invalid_argument(
        "async_emb world size changed from " + std::to_string(world_size_) + " to " +
        std::to_string(world_size)
    );
  }
  if ((world_size & (world_size - 1)) != 0) {
    throw std::invalid_argument("async_emb currently requires power-of-two EP");
  }

  rank_ = rank;
  initialized_ = false;
  std::vector<std::vector<uint8_t>> bootstrap(world_size);
  if (rank != 0) {
    return bootstrap;
  }

  ncclUniqueId id;
  ncclResult_t result = ncclGetUniqueId(&id);
  if (result != ncclSuccess) {

View on GitHub (pinned to 24c60942c5)

Solutions

1. Destroy and reconstruct AsyncEmbContext when the world size genuinely changes.
2. Ensure every rank passes the same world_size derived from one process-group initialization.
3. In elastic setups, allocate a fresh context per resize event rather than calling reset with new topology.

Example fix

// before
ctx.reset(rank, new_world_size);  // throws

// after
ctx = std::make_unique<AsyncEmbContext>(new_world_size);
ctx->reset(rank, new_world_size);

if (new_world_size != constructed_world_size) {
    // rebuild instead of reset
    ctx = std::make_unique<AsyncEmbContext>(new_world_size);
}

catch (const std::invalid_argument& e) { if rebuilt_needed -> recreate context; }

When it happens

Trigger: Calling reset(rank, n) with a different n than the AsyncEmbContext was constructed with — typically after a elastic restart where some ranks rejoined with a new world size, or when two different jobs share a context object.

Common situations: TorchElastic / torchrun restart changes world size between attempts; unit tests reusing a global context across test cases with different worker counts.

Related errors

• async_emb received an invalid rank or world size
• async_emb currently requires power-of-two EP
• async_emb context not initialized
• async_emb received an invalid NCCL bootstrap payload
• Task generators not started

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/de3ff2a40cfe9a12. Report an issue: GitHub.