xai-org/x-algorithm · error · std::invalid_argument

async_emb received an invalid rank or world size

Error message

async_emb received an invalid rank or world size

What it means

AsyncEmbContext::reset validates its rank/world_size arguments: world_size must be positive, rank must be in [0, world_size). Any violation throws invalid_argument before any NCCL work starts.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:231

    }
  }
  if (result != ncclSuccess) {
    LOG(ERROR) << "ncclCommFinalize failed: " << ncclGetErrorString(result);
    abortCommunicator("ncclCommFinalize failed");
    comm_ = nullptr;
    return;
  }

  result = ncclCommDestroy(comm_);
  if (result != ncclSuccess) {
    LOG(ERROR) << "ncclCommDestroy failed: " << ncclGetErrorString(result);
  }
  comm_ = nullptr;
}

std::vector<std::vector<uint8_t>> AsyncEmbContext::reset(int rank, int world_size) {
  if (world_size <= 0 || rank < 0 || rank >= world_size) {
    throw std::invalid_argument("async_emb received an invalid rank or world size");
  }
  if (world_size != world_size_) {
    throw std::invalid_argument(
        "async_emb world size changed from " + std::to_string(world_size_) + " to " +
        std::to_string(world_size)
    );
  }
  if ((world_size & (world_size - 1)) != 0) {
    throw std::invalid_argument("async_emb currently requires power-of-two EP");
  }

  rank_ = rank;
  initialized_ = false;
  std::vector<std::vector<uint8_t>> bootstrap(world_size);
  if (rank != 0) {
    return bootstrap;
  }

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify rank and world_size come from the same initialized process group and are 0-based rank, 1-based world size.
2. Check for swapped arguments at the call site.
3. Log rank/world_size immediately before calling reset to catch 0/-1 sentinel values.

Example fix

# before (swapped)
ctx.reset(world_size=dist.get_world_size(), rank=dist.get_rank())

# after
rank, world_size = dist.get_rank(), dist.get_world_size()
assert 0 <= rank < world_size
ctx.reset(rank, world_size)

if (world_size <= 0 || rank < 0 || rank >= world_size)
    throw std::invalid_argument("bad rank/world_size");
ctx.reset(rank, world_size);

When it happens

Trigger: Calling reset(rank, world_size) with world_size<=0, rank<0, or rank>=world_size — e.g. passing world_size where rank is expected, or using rank from a launcher that numbers ranks 1..N instead of 0..N-1.

Common situations: Argument-order swaps when wiring torch.distributed.get_rank()/get_world_size(); MPI-style 1-based ranks; world_size read as 0 because dist.init_process_group had not run yet.

Related errors

• async_emb world size changed from ${world_size_} to ${world_
• async_emb currently requires power-of-two EP
• async_emb received an invalid NCCL bootstrap payload
• XAI_ASYNC_EMB_TIMEOUT_SECONDS must be a positive integer
• type checking expression %s failed: invalid argument type: %

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/804df4dd181ecb17. Report an issue: GitHub.