xai-org/x-algorithm · error · std::invalid_argument

async_emb currently requires power-of-two EP

Error message

async_emb currently requires power-of-two EP

What it means

The internal all-to-all scheduling assumes a power-of-two expert-parallel degree, so reset() checks (world_size & (world_size-1)) == 0 and rejects non power-of-two values such as 3, 5, 6, 12. This is a documented algorithmic limitation, not a transient fault.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:240

  result = ncclCommDestroy(comm_);
  if (result != ncclSuccess) {
    LOG(ERROR) << "ncclCommDestroy failed: " << ncclGetErrorString(result);
  }
  comm_ = nullptr;
}

std::vector<std::vector<uint8_t>> AsyncEmbContext::reset(int rank, int world_size) {
  if (world_size <= 0 || rank < 0 || rank >= world_size) {
    throw std::invalid_argument("async_emb received an invalid rank or world size");
  }
  if (world_size != world_size_) {
    throw std::invalid_argument(
        "async_emb world size changed from " + std::to_string(world_size_) + " to " +
        std::to_string(world_size)
    );
  }
  if ((world_size & (world_size - 1)) != 0) {
    throw std::invalid_argument("async_emb currently requires power-of-two EP");
  }

  rank_ = rank;
  initialized_ = false;
  std::vector<std::vector<uint8_t>> bootstrap(world_size);
  if (rank != 0) {
    return bootstrap;
  }

  ncclUniqueId id;
  ncclResult_t result = ncclGetUniqueId(&id);
  if (result != ncclSuccess) {
    throw ncclError("ncclGetUniqueId", result);
  }

  std::vector<uint8_t> bytes(sizeof(id));
  std::memcpy(bytes.data(), &id, sizeof(id));
  std::fill(bootstrap.begin(), bootstrap.end(), bytes);

View on GitHub (pinned to 24c60942c5)

Solutions

1. Set expert-parallel world size to the nearest power of two (1, 2, 4, 8, 16, 32).
2. Restructure the job so the EP group is a dedicated power-of-two subgroup and other ranks are data-parallel.
3. Request node shapes that yield power-of-two total GPUs (e.g. 8-GPU nodes).

Example fix

# before
torchrun --nproc_per_node=6 train.py  # EP=6 -> throws

# after
torchrun --nproc_per_node=8 train.py  # EP=8 -> ok

bool pow2(int n) { return n > 0 && (n & (n - 1)) == 0; }
if (!pow2(ep_world_size))
    throw std::invalid_argument("EP world size must be a power of two");

bool isPow2(int n);

When it happens

Trigger: Calling reset with world_size values like 3, 5, 6, 7, 9, 12... — e.g. requesting 6 GPUs for EP instead of 4 or 8.

Common situations: Cluster allocations giving odd GPU counts; scaling experiments (12-way EP) that work in other libraries but not here; mixing data-parallel and expert-parallel ranks so the EP subgroup size is non power-of-two.

Related errors

• async_emb received an invalid rank or world size
• async_emb world size changed from ${world_size_} to ${world_
• async_emb received an invalid NCCL bootstrap payload
• Checksums internally inconsistent: Mismatch in {fn} for tens
• Checksum of key {key} in {fn} differs{extra}: 0x{cvalue:08x}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/8140d70640298d52. Report an issue: GitHub.